Family Practice Advance Access originally published online on January 29, 2007
Family Practice 2007 24(2):117-127; doi:10.1093/fampra/cml072
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Assessing the practising physician using patient surveys: a systematic review of instruments and feedback methods
a School of Medicine, Swansea University, Swansea SA2 8PP
b Department of General Practice, Centre for Health Sciences Research, Cardiff University, Neuadd Meirionnydd, Heath Park CF14 4YS
c Goldney Hall, University of Bristol, UK
Correspondence to Glyn Elwyn, Department of General Practice, Centre for Health Sciences Research, Cardiff University, Neuadd Meirionnydd, Heath Park CF14 4YS, UK; Email: elwyng{at}cardiff.ac.uk
Received 1 August 2006; Revised 16 November 2006; Accepted 4 December 2006.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Background. Individual physician performance assessment is a vital part of the medical regulation debate. In this context, the patient perspective is seen as a potentially valid component. Yet, the theoretical and empirical evidence base for such patient assessments is unclear.
Objectives. To identify and evaluate instruments designed to assess patients' experiences with an individual practising physician, and to provide performance feedback at the individual level.
Methods. Nine electronic databases were searched with no language restrictions: PubMed (1985-), Embase (1985-), PsycInfo (1985-), SIGLE (1985-), HMIC (1985-), ASSIA (1985-), CINAHL (1985-), Cochrane (1985-) and Dare (1985-).
Study selection.
Inclusion: (i) completed by patients; (ii) assess practising doctors; (iii) have capacity to assess individual doctors for performance feedback; and (iv) used for individual performance feedback.
Exclusion: (i) completed by colleagues, observers or third parties; (ii) assess medical students, nurses or non-physicians; (iii) assess purely at an organizational level; and (iv) not been used for individual feedback.
All electronic outputs were independently assessed by three reviewers. Data were extracted independently by two of three reviewers using a defined template.
Results. Six instruments met the inclusion criteria. They all combine evaluation at both organizational and individual level and implementation methods lack standardization. There is limited data on their construct validity or correlations with other attributes. The purpose and method of individual feedback are not well specified, and the evidence to date about the effectiveness of feedback to obtain improvement indicates professional resistance.
Conclusions. For formative goals, more clarity is needed about the aim of providing patient assessments feedback to individual doctors: ‘who’ should do it and ‘how’ to do so to best effect. We need to know whether feedback improves doctor performance and how these evaluations correlate with other physician attributes. For summative purposes more research is required on validity and reliability.
Keywords. Patient surveys of clinician performance, performance assessment, feedback of patient assessment.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Physicians unlike the children of Keillor's1 Lake Wobegon are not all above average. There is indeed concern that substandard practice exists and that physician performance needs to be measured regularly. However, there is little, if any, consensus on how the performance of practising physicians should be measured and by whom—by other doctors, by patients or by independent third parties. Our recent review concluded that the existing methods of assessment by peers lack proven validity,2 although there is a recognition that doctors ‘know’ the identity of poor performers.3 Nevertheless, formalizing the assessment of physicians' performance has assumed some urgency with a focus on how we should make valid and reliable judgements. As we found, the validity of peer-based judgements is open to question.2 What then is the role of patient-based assessments, given that they experience the performance of doctors at first hand? In the UK, a contractual incentive stimulated general practitioners to obtain data using recently developed and recommended patient survey instruments,4–6 while in the US the Maintenance of Certification (MOC) program7,8 is encouraging use of confidential patient satisfaction measures as one element of assessing performance in practice.9 As the regulation debate continues10–14 in both countries and is mirrored internationally, it is important to assess the science underpinning these instruments in more detail.
The patient view began to emerge as a potential means of feedback for service improvement from the 1980s15–17 drawing on consumer survey literature from the previous 20 years. Yet, it is only in the last 5 years that the potential for educational feedback at the level of an individual practitioner has come into focus.18–20 The need for valid and reliable performance feedback to assess fitness to practice21 adds another focus to this patient source. The increasing importance of the patient view as part of a ‘360-degree assessment’22 warrants a review of available instruments and their potential contribution to performance assessment.
We sought to identify and evaluate existing instruments for patient assessment of their experience with an individual doctor in routine practice that have been used for feedback to individual doctors, to evaluate their development, validity and reliability, as well as their potential to identify poor performance of the individual doctor for further assessment.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Search strategy
A systematic search was performed for published material using Medical Subject Headings and text words in the following nine electronic databases: PubMed (1985-), Embase (1985-), PsycInfo (1985-), SIGLE (1985-), HMIC (1985-), ASSIA (1985-), CINAHL (1985-), Cochrane (1985-) and Dare (1985-). Initial searches identified little work on patient surveys at the level of individual episodes of care prior to 198523 where the focus was at the level of organization or service. The search strategy used for all the databases used combinations of text words and MeSH terms found to be relevant from pilot searches; there were no language restrictions. The text word combinations used were the following: ‘patient satisfaction’ or ‘patient participation’ or ‘physician-patient relations’; ‘physician’ or ‘doctor’; and ‘questionnaire’ or ‘survey’ or ‘instrument’ or ‘feedback’ or ‘evaluation’. For PubMed and Embase a further combination of ‘Professional competence’ was added as text and MeSH term to narrow the search for these databases. Reference lists from review articles and included papers were searched for additional instruments. The initial search was run in July 2004 and rerun November 2005.
Inclusion and exclusion criteria
To be included, the instruments needed to be published in the peer-reviewed literature and fulfil the following criteria:
- (i) be completed by patients;
- (ii) assess practising doctors;
- (iii) have capacity to assess individual doctors for performance feedback; and
- (iv) have been used for individual performance feedback.
- (ii) assess practising doctors;
Instruments were excluded if any of the following were evident:
- (i) completed by colleagues, observers or third parties;
- (ii) they assessed medical students, nurses or non-physicians;
- (iii) they assessed purely at an organizational level; and
- (iv) they had not been used for individual feedback.
- (ii) they assessed medical students, nurses or non-physicians;
Selection
Three reviewers (RGE, SE and BE) assessed all electronic outputs (abstracts) independently. Full-text articles considered as possibly relevant were assessed by three reviewers (RGE, GE and AE). Selection against the criteria was then decided by the reviewers together. Where it was unclear from the literature whether the instrument had the capacity to assess an individual, the authors were contacted for further information. The decision on final inclusion against criterion 4 was made on the basis of the published data backed up by email contact with all the authors to identify any missed data from the peer-reviewed literature. This included asking the authors to check whether we had omitted any instruments from the list.
Data extraction and synthesis
A template was piloted and amended for data extraction. The template covered the following areas: conceptual basis for the instrument's development, the measurement construct, piloting and validation process, characteristics of patient and physician sample, psychometric data, setting of main use and description of feedback process. One researcher (RGE) applied the template to all included studies; a second researcher (GE or AE) also undertook data extraction for each instrument. The data were summarized and a synthesis conducted by RGE, GE and AE.24
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Figure 1 summarizes the results of the search.
|
We identified 3476 studies from the search strategy on nine databases (SIGLE, HMIC, PsycInfo as one database bundle 3937, ASSIA 831, CINAHL 649, Cochrane 265, Dare 1367, Embase 245 and PubMed 1432) after the two search outputs were combined and duplicates removed.
In summary, 119 papers were retrieved for full-text assessment; 55 were excluded by a single reviewer (RGE) with no instrument being described. From the remaining 64 papers we identified 25 potential instruments. Citation follow-up identified a further nine instruments, making a total of 34 instruments for detailed assessment. Eleven instruments were excluded because they did not have an explicit aim to provide data for individual physician feedback, 1 lacked psychometric data and 16 were excluded because there was no evidence that they had been used to provide individual feedback. Six instruments met all inclusion criteria.
Instruments excluded
No focus on individual performance.
The 11 instruments lacking an explicit aim to provide data for individual physician feedback are listed as follows: Primary Care Assessment Survey,25 Practice Accreditation and Improvement Survey,26 Consumer Assessment of Health Providers and Systems,27,28 Components of Primary Care Index,29 Quality from Patients Perspective,30 Patient Satisfaction Questionnaire,31 Parents Perceptions of Primary Care Measure32 and questionnaires by Bitzer et al.,33 Gericke et al.,34 Bredart et al.35 and Etter and Perneger.36 For the Visit-Specific Questionnaire and Patient Satisfaction Index37 psychometric data were unobtainable and excluded at this stage, making 12 excluded in total.
Not used for individual performance feedback. The 16 instruments excluded because there was no evidence as yet that they had been used for providing individual feedback are as follows: Consultation and Relational Empathy Measure (CARE),38,39 Patient Enablement Instrument (PEI),40–44 Consultation Quality Index (CQI),41 Consultation Quality Index-2 (CQI-2),45 Patient-Doctor Relationship Questionnaire,46 Medical Interview Satisfaction Scale,47,48 Patient Experience Questionnaire,49,50 Medical Outcomes Study Questionnaire,51 Patient Intentions Questionnaire/Expectations Met Questionnaire,52 Physicians' Humanistic Behaviours Questionnaire,53 Patient-Doctor Interaction Scale54 and unnamed instruments developed by authors Crossley et al.,55 Haddad et al.,56 Wooliscroft and Howell,57 Matthews and Feinstein16 and Feletti et al.15 Many of these 16 excluded instruments do not appear to have been developed beyond their initial research findings, perhaps highlighting a lack of interest in their implementation and associated costs. The PEI40–44 evaluates the doctor as being more or less ‘enabling’ in terms of the patient's well-being and coping status and has also been combined with data on length of consultation and how well the patient knows the doctor in the CQI.41 However, it is essentially a patient-based outcome measure58 rather than an evaluation of the patient's experience of visiting the doctor and does not have specific data regarding experience of its use in feedback to individual doctors. The CARE instrument developed by Mercer38,39 aims to give individual physicians direct feedback on their relational empathy as perceived by the patient and is being piloted as part of a planned new licensing examination in the UK.59 The CQI and CARE have now been combined as the CQI-2.45
Instruments included
Six instruments with clear description of development and validation and which have been used for physician feedback are included17,60–63; these are the Consultation Satisfaction Questionnaire (CSQ),17,64,65 the Chronically Ill Patients Evaluate General Practice (CEP),60 the General Practice Assessment Questionnaire (GPAS, now GPAQ),62 the Patients Evaluate General/Family Practice (EUROPEP),61 Improving Practice Questionnaire (IPQ)63 and the Provider-Specific Survey (PSS).66,67 Details about the conceptual basis, development process, psychometric evaluation and how assessments from these instruments have been used for feedback are contained in Tables 1–3.
|
|
|
The six instruments which meet the inclusion criteria are as follows.
Consultation Satisfaction Questionnaire. The 18-item CSQ was developed by Baker in 1990.17,64,68,70,79 It is a quantitative assessment of patient satisfaction from a specific visit with a specific doctor in relation to four domains: general satisfaction, professional care, depth of relationship and perceived time. A separate tool was developed for evaluating the organizational aspects, namely the Surgery Satisfaction Questionnaire.
Chronically Ill Patients Evaluate General Practice. The CEP, developed by Wensing et al.,19,60 is a 51-item questionnaire for evaluating 10 dimensions of care; it has a mix of organizational- and practitioner-level evaluation items (appointments/emergencies, premises, continuity, cooperation, medical care, competency/accuracy, relation and communication, privacy, information and advice and support). Providing individual feedback to doctors did not lead to any improvement on the doctors' communication behaviour, and many doctors perceived the patient views as less relevant than before the study.19
General Practice Assessment Questionnaire. The GPAS, now GPAQ, is a 53-item questionnaire for evaluating five areas of care: access, technical competence, effective GP communication, GP interpersonal abilities and effective organization of care.62,73,80,81 It has a mix of organizational and ‘usual doctor’ evaluation which is not necessarily visit specific. The shorter 36-item GPAQ has a postal version which is not visit specific and a post-consultation version which is visit specific.
EUROPEP. The EUROPEP instrument is a 23-item questionnaire for evaluating five dimensions of care: relation and communication, medical care, information and support, continuity and cooperation, access and facilities.61,72,75 It has a mix of organizational and ‘usual’ practitioner evaluation that is not visit specific but is to be considered ‘over the last 12 months’.
Improving Practice Questionnaire. The IPQ is a 27-item questionnaire combining evaluation of two components of care: the Doctor's Interpersonal Skills Questionnaire (DISQ) and service issues of access and environment.18,20,26,63,76,82–85 It has a mix of organizational and individual visit-specific evaluations.
Provider-Specific Survey. The PSS is a 34-item scale combining a visit-specific evaluation of five aspects of care: making the appointment, arrival at the office and registration, waiting to see the clinician, service by staff other than the main clinician and seeing the clinician.66,67
Synthesis
Although there are several instruments that use patient evaluations, the aim of the individual feedback from these instruments is difficult to discern beyond a general intention to measure ‘improvement’ along the scales of the items being rated. There are none that clearly state an aim to use patient assessments of individual doctors to provide direct summative judgements against a standard and there is usually an implied use for formative or developmental performance improvement. Those that actually provide individual assessments have only recently been used in this way, and data on evaluating the effect of feedback were limited to two instruments CEP and IPQ.19,77,78 How the feedback process is conducted, by whom, to whom and in what setting are not specified. Methods of feedback to individuals are unspecified, beyond reflecting on the normative data, with or without the help of a colleague. The implicit assumption seems to be that normative comparisons will motivate change in professionals because it is believed they are sensitive to patient views about their performance. However, clinician reaction to the feedback of patients' assessments has not been positive.19 Although there are less than a handful of studies on this issue, there is evidence that clinicians can be defensive and resistant, unless feedback is provided during early training grades.18,20 These limited data suggest that the setting and feedback process are crucial to achieve any effective change for doctors in practice.
Five of six included instruments have been developed and evaluated in European primary care and one US instrument in office-based primary and specialist care. A common theme with all instruments, except the CSQ,17 is their attempt to evaluate at both the organizational and individual level in a single instrument. While explicit labelling of items helps differentiate these aspects, there is scope for crossover bias and potential for patients to be unclear about whether they are primarily evaluating the organization or an individual clinician. Another source of potential confusion is the variation between different methods of implementation such as immediate post-consultation use and postal survey with consequent variations in score. Sometimes more than one implementation method is used for the same instrument. Different implementation gives rise to different evaluations, which will undermine benchmarking and comparative feedback.
There are very little data on how patient scores correlate with other assessments of a practising physician, suggesting a need for more work on construct validity. A very modest correlation was found between the DISQ scores and expert assessment of training grade doctors,83 while no correlation was found between the CSQ and performance in video assessment in the Membership of Royal College of General Practioners exam.86 This clearly needs further research if data were to be used for performance management or assessment of fitness to practice.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Six validated instruments met the criteria for patient assessment of their experience with an individual physician and which have been used for feedback to individual physicians. There is no clear statement of purpose for each instrument as a formative or summative assessment. All were developed in primary care or outpatient setting and appear to have the potential for identifying poorly performing doctors. However, there are as yet few data on which this could be based with confidence and only the PSS explicitly attempts to address dealing with low scores. Other than the early CSQ, all the instruments have varying methods of implementation and include evaluations of the organization or service alongside individual evaluation, leading to difficulty in interpreting their findings.
The purpose for assessing must be clear. To be used in a summative way for decision making, the science underpinning the instruments needs to be more robust. The construct validity of these instruments in terms of correlation with other assessment perspectives needs further research. The lack of any meaningful correlation between one form of assessment (the video component of the MRCGP) and patient feedback via the CSQ or PEI highlights this.86 If the purpose is formative, then we need a better scientific understanding of the effect of feedback and how to best use it.
The evidence of the effect of feedback on practising physicians is limited to two of the five instruments19,77,78,87 where no improvement was shown and a mixed reaction from doctors, one study generating negative perceptions in the feedback group.19,87 The only evidence of positive effect relates to the DISQ element of the IPQ when used in a structured educational setting with GP registrars early in their training with less effect seen later in training.18,20 The insufficient evidence does not allow more comment than that the effect of feedback can be equivocal or negative.
Strengths and weakness
We used several databases with delineated search strategies and complementary methods to identify relevant material. However, the search strategy was narrower for the PubMed and Embase databases and non-English language sources were not specifically sought, each potentially limiting recall. Multiple assessments were undertaken at key decision points and dual independent data extraction methods were applied. We focussed our research question on a current issue for the debate around the assessment of practising doctors and we are unaware of any similar analysis.
Current context
Quality improvement initiatives classically intervene at the system level to enhance quality and safety in health care. Continuing professional development and appraisal also examine how the individual clinician is performing. In this latter sphere, increasing importance is given to patients' views of their doctors. In UK primary care, for example these assessments have already been integrated into the incentivized ‘Quality and Outcomes Framework’ of doctors' contract with the National Health Service5 and form part of the UK Chief Medical Officers policy proposals,14 while in the US the MOC program8,9,88 is being developed on a continuous cycle. In this way, patient evaluations are now becoming part of the landscape of clinical care development—the 360-degree assessment22 that includes patient, peer and other staff evaluations; clinical or health outcome indicators and identifying cost-effective and evidence-based practice, all focussed on the individual clinician.
Implications
We suggest that greater clarity is needed about the purposes to which the available instruments may be put. The overall aim of these instruments seems to be formative in identifying the areas of consulting behaviour from patient experience, where there is scope to improve in relation to the patient-centred model of consulting.89 They rely on normative comparisons to motivate change in professionals who are assumed to be sensitive to patient views on their behaviour. Relying on this is questionable as evidence from wider organizational research suggests that such normative comparisons can lead to performance decline as well as improvement.90 There are variable methods for administration and sampling, and policies on data ownership and benchmarking are uncertain. Individual-level feedback methods are also not specified beyond reflecting on the normative data (with or without a colleague), whereas potential responses to feedback in terms of what doctors can do to improve communication skills are described copiously. There is a gap between data for feedback and action to improve, where the precise mechanism of formative individual feedback for clinicians is unclear. Research is particularly required on how to undertake effective feedback processes.
Alternatively, the goal may be summative. This would require a major advance in the validation of the instruments, with further assessment of construct validity in current settings (usually primary care), validation in others settings such as secondary care and analysis of the determinants of variation. In this context, the aim would be to look for outliers who would then be subjected to further assessment of competence or in the extreme scenario, their fitness to practise. Holders of benchmark data would need to consider their response to outliers, and whether they are obliged to disclose about possible poor performers. Responsiveness would also need validating in this high-stakes context where detectable and demonstrable improvement needs to be reliably measured.
|
Conclusions |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Patients are the end-users of health care. Clinicians have been lukewarm towards feedback from users,19,78 but this cannot be dismissed. We must continue to measure patients' assessments of their experience with individual clinicians and try to understand more what they mean, how they correlate with other aspects of individual performance, and how doctors can learn and improve by the assessment. If, however, these assessments become more summative, part of ‘performance management’, and integral to re-certification, then in terms of procedural justice91 there has to be more science around validity, reliability, standardized administration of the instruments and sampling and investment in comparative benchmarking. We also need to know more about how to use these assessments, who should provide the feedback and which process works best. The available instruments offer a platform from which to develop patients' assessments of physician performance, as patients are best placed to make judgements about many crucial areas of practice.
|
Declaration |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
Funding: None.
Ethical approval: None.
Conflicts of interest: None.
|
Notes |
|---|
Evans RG, Edwards A, Evans S, Elwyn B and Elwyn G. Assessing the practising physician using patient surveys: a systematic review of instruments and feedback methods. Family Practice 2007; 24: 117–127.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionConclusionsDeclarationReferences |
|---|
1 Keillor G. Lake Wobegon Days (1985) New York: Antheneum.
2 Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians. Br Med J (2004) 328:1240–1243.
3 Pringle M. Re-evaluating revalidation and appraisal. Br J Gen Pract (2003) 53:437–438.[ISI][Medline]
4 Client-Focused Evaluations Program. Improving Practice Questionnaire (2003) CFEP-UK Surveys, University of Exeter, Exeter. www.cfep.net. Accessed July 2006.
5 General Practitioners Committee BMA and the NHS Confederation. New GMS Contract 2003 (2003) London: Investing in General Practice.
6 National Primary Care Research and Development Centre. GPAQ—A Questionnaire for the GP Contract (2003) Manchester University, Manchester. www.gpaq.info. Accessed July 2006.
7 Lipner R, Blank L. The value of patient and peer ratings in recertification. Acad Med (2002) 77:S64–S66.[CrossRef][ISI][Medline]
8 American Board of Medical Specialties. Maintenance of Certification. American Board of Medical Specialities, Evanston. www.abms.org. Accessed July 2006.
9 Batmangelich S, Adamowski S. Maintenance of certification in the United States: a progress report. J Contin Educ Health Prof (2004) 24:134–138.[CrossRef][Medline]
10 Lipner R, Bylsma W, Arnold G, Fortna G, Tooker J, Cassel C. Who is maintaining certification in internal medicine—and why? A national survey 10 years after initial certification. Ann Intern Med (2006) 144:29–36.
11 Lipner R. The value of physician assessment. Ann Intern Med (2006) 144. Annals online, 3 Mar 2006.
12 Lakhani M. GMC and the future of revalidation: a way forward. Br Med J (2005) 330:1326–1328.
13 Pringle M. Revalidation of Doctors: the Credibility Challenge (2005) London: The Nuffield Trust.
14 Donaldson L. Good Doctors, Safer Patients: Proposals to Strengthen the System to Assure and Improve the Performance of Doctors and to Protect the Safety of Patients (2006) Department of Health.
15 Feletti G, Firman D, Sanson-Fisher R. Patient satisfaction with primary care consultations. J Behav Med (1985) 9:389–399.[CrossRef][ISI]
16 Matthews D, Feinstein A. A new instrument for patients' ratings of physician performance in the hospital setting. J Gen Intern Med (1989) 4:14–22.[ISI][Medline]
17 Baker R. Development of a questionnaire to assess patients' satisfaction with consultations in general practice. Br J Gen Pract (1990) 40:487–490.[ISI][Medline]
18 Greco M, Brownlea A, McGovern J. Impact of patient feedback on the interpersonal skills of general practice registrars: results of a longitudinal study. Med Educ (2001) 35:748–756.[CrossRef][ISI][Medline]
19 Wensing M, Vingerhoets E, Grol R. Feedback based on patient evaluations: a tool for quality improvement? Patient Educ Couns (2003) 51:149–153.[CrossRef][ISI][Medline]
20 Greco M, Francis W, Buckley J, Brownlea A, McGovern J. Real-patient evaluation of communication skills teaching for GP registrars. Fam Pract (1998) 15:51–57.
21 General Medical Council. Good Medical Practice (2001) GMC, London.
22 King J. 360 degree appraisal. Br Med J Career Focus (2002) 324:S195.
23 Linder-Pelz M, Struening E. The multidimensionality of patient satisfaction with a clinic visit. J Community Health (1985) 10:42–54.[CrossRef][Medline]
24 Slavin RE. Best evidence synthesis: an intelligent alternative to meta-analysis. J Clin Epidemiol (1995) 48:9–18.[CrossRef][ISI][Medline]
25 Safran DG, Kosinski M, Tarlov AR. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care (1998) 36:728–739.[CrossRef][ISI][Medline]
26 Greco M, Sweeney K, Brownlea A, McGovern J. The Practice Accreditation And Improvement Survey (PAIS): what patients think. Aust Fam Physician (2001) 30:1096–1100.[Medline]
27 Darby C, Crofton C, Clancy C. Consumer Assessment of Health Providers and Systems (CAHPS): evolving to meet stakeholder needs. Am J Med Qual (2006) 21:144–147.
28 Hays R, Chong K, Brown J, Spritzer K, Horne K. Patient reports and ratings of individual physicians: an evaluation of the DoctorGuide and Consumer Assessment of Health Plans Study provider-level surveys. Am J Med Qual (2003) 18:190–196.[Abstract]
29 Flocke SA. Measuring attributes of primary care: development of a new instrument. J Fam Pract (1997) 45(1):64–74.[ISI][Medline]
30 Narthorst-Boos J, Munck I, Eckerlund I. An evaluation of the QSP and the QPP: two methods for measuring patient satisfaction. Intern J Qual Healthcare (2001) 13:257–264.[CrossRef]
31 Grogan S, Conner M, Willits D, Norman P. Development of a questionnaire to measure patients' satisfaction with general practitioners' services. Br J Gen Pract (1995) 45:525–529.[ISI][Medline]
32 Seid M, Varni J. Measuring primary care for children of Latino farmworkers: reliability and validity of the Parent's Perceptions of Primary Care Measure (P3C). Matern Child Health J (2005) 9:49–57.[CrossRef][ISI][Medline]
33 Bitzer E, Dierks M, Dorning H, Schwartz F. Patient satisfaction with ambulatory care physicians—psychometric testing of a standardized questionnaire. Z Gesundheitswiss (1999) 7:196–209.
34 Gericke C, Schiffhorst G, Busse R, Haussler B. [A validated questionnaire for measuring patient satisfaction in general and specialist ambulatory medical care: the Qualiskope-A]. Gesundheitswesen (2004) 66:723–731.[CrossRef][ISI][Medline]
35 Bredart A, Razavi D, Delvaux N, Goodman V. A comprehensive assessment of satisfaction with care for cancer patients. Support Care Cancer (1998) 6:518–523.[CrossRef][ISI][Medline]
36 Etter J, Perneger T. Validating a satisfaction questionnaire using multiple approaches: a case study. Soc Sci Med (1997) 45:879–885.[CrossRef][ISI][Medline]
37 Yancy W, Macpherson D, Hanusa B. Patient satisfaction in resident and attending ambulatory care clinics. J Gen Intern Med (2001) 2001:755–762.
38 Mercer S, Maxwell M, Heaney D. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract (2004) 21:699–705.
39 Mercer S, McConnachie A, Maxwell M. Relevance and practical use of the Consultation and Relational Empathy (CARE) measure in general practice. Fam Pract (2005) 22:328–334.
40 Howie JGR, Heaney DJ, Maxwell M, Walker JJ. A comparison of a Patient Enablement Instrument (PEI) against two established satisfaction scales as an outcome measure of primary care consultations. Fam Pract (1998) 15:165–71.
41 Howie JGR, Heaney DJ, Maxwell M, Walker JJ, Freeman GK. Developing a ‘consultation quality index’ (CQI) for use in general practice. Fam Pract (2000) 17:455–461.
42 Howie JGR, Heaney DJ, Maxwell M. Quality, core values and the general practice consultation: issues of definition, measurement and delivery. Fam Pract (2004) 21:458–468.
43 Howie JGR, Heaney DJ, Maxwell M. Measuring quality in general practice: pilot study of a needs, process and outcome measure. R Coll Gen Pract Occas Pap (1997) 75.
44 Howie JGR, Heaney DJ, Maxwell M. Quality at general practice consultations: cross-sectional survey. Br Med J (1999) 319:738–743.
45 Mercer S, Howie J. CQI-2-a new measure of holistic interpersonal care in primary care consultations. Br J Gen Pract (2006) 56:262–268.[ISI][Medline]
46 Van der Feltz-Cornelis CM, Van Oppen P, Van Marwijk H. A patient-doctor relationship questionnaire (PDRQ-9) in primary care: development and psychometric evaluation. Gen Hosp Psychiatry (2004) 26:115–120.[CrossRef][ISI][Medline]
47 Meakin R, Weinman J. The ‘Medical Interview Satisfaction Scale’ (MISS-21) adapted for British general practice. Fam Pract (2002) 19:257–263.
48 Wolf MH, Putnam SM, James SA. The medical interview satisfaction scale: development of a scale to measure patient perceptions of physician behaviour. J Behav Med (1978) 1:391–401.[CrossRef][Medline]
49 Steine S, Finset A, Laerum E. A new, brief questionnaire (PEQ) developed in primary health care for measuring patients' experience of interaction, emotion and consultation outcome. Fam Pract (2001) 18:410–418.
50 Petterson K, Veenstra M, Guldvog B, Kolsatd A. The Patient Experiences Questionnaire: development, validity and reliability. Intern J Qual Health care (2004) 16:453–463.[CrossRef]
51 Stump TE, Dexter PR, Tierrey WM. Measuring patient satisfaction with physicians among older and diseased adults in primary care municipal outpatient setting: an examination of three instruments. Med Care (1995) 33(9):958–972.[ISI][Medline]
52 Williams S, Weinman J, Dale J, Newman S. Patient expectations: what do primary care patients want from the GP and how far does meeting expectations affect patient satisfaction. Fam Pract (1995) 12:193–201.
53 Weaver MJ, Ow CL, Walker DJ. A questionnaire for patients' evaluations of their physicians' humanistic behaviours. J Gen Intern Med (1993) 8(3):135–139.[ISI][Medline]
54 Smith JK, Falvo D, McKillip J. Measuring patient perceptions of the patient-doctor interaction. Eval Health Prof (1984) 7:77–94.
55 Crossley J, Eiser C, Davies H. Children and their parents assessing the doctor-patient interaction: a rating system for doctors' communication skills. Med Educ (2005) 39:820–828.[CrossRef][ISI][Medline]
56 Haddad S, Potvin L, Roberge D. Patient perception of quality following a visit to a doctor in a primary care unit. Fam Pract (2000) 17:21–29.
57 Woolliscroft J, Howell J. Resident-patient interactions: the humanistic qualities of internal medicine residents assessed by patients, attending physicians, program supervisors and nurses. Acad Med (1994) 69:216–224.[ISI][Medline]
58 Fitzpatrick R. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess NHS R&D HTA programme (1998) 1–72.
59 Wass V. A guide to the new MRCGP exam. Br Med J Careers (2005) 331:158–159.
60 Wensing M, Grol R, Van Weel C. Quality assessment using patients' evaluations of care. Eur J Gen Pract (1998) 4:150–153.
61 Grol R, Wensing M. Patients Evaluate General/Family Practice: the EUROPEP Instrument: EQuiP: the Task Force on Patient Evaluations of General Practice Care (2000) Centre for Quality Care Research, Nijmegen.
62 Ramsay J, Campbell J, Schroter S, Green J, Roland M. The General Practice Assessment Survey (GPAS): tests of data quality and measurement properties. Fam Pract (2000) 17:372–379.
63 Greco M, Powell R, Sweeney K. The Improving Practice Questionnaire (IPQ): a practical tool for general practices seeking patient views. Educ Prim Care (2003) 14:440–448.
64 Baker R. The reliability and criterion validity of a measure of patients' satisfaction with their general practice. Fam Pract (1991) 8:171–177.
65 Baker R, Whitfield M. Measuring patient satisfaction: a test of construct validity. Qual Health Care (1992) 1:104–109.[Abstract]
66 Nelson E, Gentry M, Mook K, Spritzer K. How many patients are needed to provide reliable evaluations of individual clinicians? Med Care (2004) 42:259–266.[CrossRef][ISI][Medline]
67 Epstein KR, Laine C, Farber N, Nelson E, Davidoff F. Patients' perceptions of office medical practice: judging quality through the patients' eyes. Am J Med Qual (1996) 11:73–80.
68 Baker R. Characterisitics of practices, general practitioners and patients related to levels of patients' satisfaction with consultations. Br J Gen Pract (1996) 46:601–605.[ISI][Medline]
69 Baker R. Pragmatic model of patient satisfaction in general practice: progress towards a theory. Qual Health Care (1997) 6:210–204.
70 Baker R, Streatfield J. What type of general practice do patients prefer? Exploration of practice characteristics influencing patient satisfaction. Br J Gen Pract (1995) 45:654–659.[ISI][Medline]
71 Bower P, Roland MO. Bias in patient assessments on general practice: General Practice Assessment Survey scores in surgery and postal responders. Br J Gen Pract (2003) 53:126–128.[ISI][Medline]
72 Bower P, Mead N, Roland M. What dimensions underlie patient responses to the general practice assessment survey? A factor analytic study. Fam Pract (2002) 19:489–95.
73 Bower P, Roland M, Campbell J, Mead N. Setting standards based on patients views on access and continuity: secondary analysis of data from the general practice assessment survey. Br Med J (2003).
74 Grol R, Wensing M, Mainz J. Patients' priorities with respect to general practice care: an internal comparison. European Task Force on Patient Evaluations of General Practice (EUROPEP). Fam Pract (1999) 16:4–11.[Medline]
75 Wensing M, Jung H, Mainz J. Which aspects of general practice care are important for patients? A systematic literature review. Soc Sci Med (1998) 47:1573–1588.[CrossRef][ISI][Medline]
76 Greco M, Carter M. Improving Practice Questionnaire (IPQ) Tool Kit (2002) Chichester: Aeneas Press.
77 Greco M, Cavanagh M, Brownlea A, McGovern J. Validation studies of the Doctors' Interpersonal Skills Questionnaire. Educ Gen Pract (1999) 10:256–264.
78 Greco M, Spike N, Powell R, Brownlea A. Assessment of GP registrars' communication skills: a comparison of patient and GP examiner ratings. Med Educ (2002) 36:366–376.[CrossRef][ISI][Medline]
79 Greco M, Brownlea A, McGovern J, Cavanagh M. Consumers as educators: implementation of patient feedback in general practice training. Health Commun (2000) 12:173–93.[CrossRef][ISI][Medline]
80 Al-Shawi A, MacEachern A, Greco M. Patient assessment of surgeons' interpersonal skills. A tool for appraisal and revalidation. Clin Govern Int J (2005) 10:212–216.[CrossRef]
81 Carter M, Greco M. Impact of systematic feedback on general practices, staff, patients and primary care trusts. Educ Prim Care (2004) 15:30–38.
82 Greco M, Carter M. Does a patient survey make a difference? Educ Prim Care (2004) 15:183–189.
83 McKinstry B, Walker J, Blaney D, D H. Do patients and expert doctors agree on the assessment of consultation skills? Fam Pract (2004) 21:75–80.
84 Vingerhoets E, Wensing M, Grol R. Feedback of patients' evaluations of general practice care: a randomised trial. Qual Health Care (2001) 10:224–228.
85 American Board of Internal Medicine. Recertification: Program for Continuous Professional Development. www.abim.org/cpd.
86 Stewart M, Brown J, Weston W. Patient-Centred Medicine: Transforming the Clinical Method (2003) 2nd edn. Oxford: Radcliffe Medical Press.
87 DeNisi A, Kluger A. Feedback effectiveness: can 360-degree appraisals be improved? Acad Manage Exec (2000) 14:129–139.
88 van den Bos K. Procedural and distributive justice: what is fair depends more on what comes first than on what comes next. J Pers Soc Psychol (1997) 72:95–104.[CrossRef][ISI]
89 Poulton B. Use of the consultation satisfaction questionnaire to examine patients' satisfaction with general practitioners and community nurses: reliability, replicability and discriminant validity. Br J Gen Pract (1996) 46:26–31.[ISI][Medline]
90 Kinnersley P, Stott N, Peters T. A comparison of methods for measuring patient satisfaction with consultations in primary care. Fam Pract (1996) 13:41–51.
91 Campbell S, Hann M, Hacker J. Identifying predictors of high quality care in English general practice: observational study. Br Med J (2001) 323:1–6.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  G. Elwyn, S. Buetow, J. Hibbard, and M. Wensing Respecting the subjective: quality measurement from the patient's perspective BMJ, November 17, 2007; 335(7628): 1021 - 1022. [Full Text] [PDF]
|
|
E-letters:
Read all E-letters
- PEI, CARE and CQI-2
- John GR Howie, et al.
- Family Practice, 12 Aug 2007 [Full text]
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||