Family Practice Advance Access originally published online on April 26, 2006
Family Practice 2006 23(4):437-443; doi:10.1093/fampra/cml017
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Patient satisfaction with large-scale out-of-hours primary health care in The Netherlands: development of a postal questionnaire
a Department of General Practice, Academic Medical Centre—University of Amsterdam Meibergdreef 15, 1105 AZ Amsterdam
b Centre for Quality of Care Research, Radboud University Nijmegen WOK 229, PO Box 9101, 6500 HB Nijmegen, The Netherlands
Correspondence to Eric Moll van Charante; Email: e.p.mollvancharante{at}amc.uva.nl.
Received 4 July 2005; Accepted 21 March 2006.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Background. Since the turn of the millennium, out-of-hours primary health care in The Netherlands has faced a substantial change from small locum groups towards large GP cooperatives. Improving the quality of care requires evaluation of patient satisfaction.
Objective. To develop a reliable postal questionnaire for wide-scale use by patients contacting their out-of-hours GP cooperative and to present the results of a national survey.
Methods. Literature review and interviews with both patients and health carers were carried out to identify issues of potential relevance, followed by two postal pilot studies and additional interviews to remove or rephrase items. Finally, postal questionnaires were sent to 14 400 people who contacted one of 24 GP cooperatives in The Netherlands.
Results. Overall response was 52.2% for all types of contact. Three scales were identified prior to the field phase and confirmed by principal components analysis: telephone nurse, doctor and organization. Reliability was high, with Cronbach's alphas and intraclass correlation coefficients exceeding 0.70 for all scales. Only items in the organization scale showed clear differences among the participating cooperatives. Respondents receiving telephone advice showed lower levels of satisfaction than respondents with other types of contact (P < 0.001); centre consultation scored lower than home visit (P < 0.030 or less for all differences).
Conclusion. A reliable measure of patient satisfaction has been developed that can also be used for the comparison of GP cooperatives on an organizational level. Overall satisfaction was high, showing highest levels for home visit and lowest levels for telephone advice.
Keywords. Family practice, out-of-hours, patient satisfaction, primary health care.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Since the turn of the millennium, Dutch GPs have reorganized their out-of-hours primary health care substantially, following examples in the UK and Denmark.1,2 Due to feelings of increasing and inappropriate demand, fatigue and job dissatisfaction, as have been described elsewhere,3–5 the decision was made to set up large-scale GP cooperatives. These organizations replaced most of the small locum groups in which GPs had been used to provide care for the local population. In general, out-of-hours care shifted from care by a familiar GP in the vicinity towards more centralized care provided by a cooperative further away. Currently, around 120 GP cooperatives serve more than 90% of the total Dutch population (16.3 million people). The number of full-time GPs participating in these services generally ranges from 40 to 120 with patient populations between 80 000 and 500 000 people. Supervised by the GPs, nurses perform the telephone triage and decide whether they advise the patients themselves, plan a consultation with the GP in the cooperative or recommend the GP to make a home visit. Satisfaction of patients' legitimate demands is a major objective of all medical care, but is also recognized as one of the possible outcome measures of quality of care.6 Several attempts have been made to evaluate patients' views on this new out-of-hours primary health care provision,7–17 yet in view of possible cultural and organizational differences, its validity for the Dutch situation had still to be assessed.
Furthermore, the increasing demand for benchmarking quality of care calls for the development of a valid and reliable measure of patient satisfaction that can both aid individual GP cooperatives in improving their quality of care and also be of use for a nationwide comparison.
The main objectives of this study were the development of a reliable postal questionnaire for wide-scale use of patients contacting their GP out-of-hours cooperative and to present the first results of a national survey.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Questionnaire development
Reviewing the literature (phase 1). It was decided to depart from a literature review, since McKinley et al.13 concluded earlier, that their extensive work on identifying relevant items for evaluating out-of-hours primary health care through the use of focus groups had only yielded a few new items to the literature.
Medline was searched with a combination of the terms ‘general practitioner’, ‘patient satisfaction’ and ‘out of hours’. In total, 34 mainly British articles were found. Three unpublished questionnaires evaluating out-of-hours primary care from different Dutch Departments of General Practice were also studied. This way, an item bank with potential questions on all three types of contact with the GP cooperative was developed.
Interviewing the parties involved (phase 2). Eight GPs and four telephone nurses were invited to review the questions and focus on items with the potential for improving quality of care. They added a few items on the telephone triage and continuity of care and proposed a few open questions to leave room for additional qualitative remarks.
Three experts in the field of questionnaire development were each asked independently to comment on various clinimetrical aspects of the first concept questionnaire. Their most important suggestion was to split the questionnaire into three separate ones for telephone advice, centre consultation and home visit, since each questionnaire partly addressed different issues.
A panel of six patients from a regional patient federation was asked to study the concept questionnaire, to comment on the items' relevance and phrasing and to indicate whether they had any additional relevant items. They appeared to have a strong preference for a functional, rather than a more random ordering of the items, linked to the telephone nurse, the doctor and the organization, respectively. They found items that were either worded positively or negatively to be confusing and overabundant when addressing the same issues. Instead of the proposed 7-point Likert scale they suggested to use a 10-point scale, similar to the widely used grading system in Dutch primary and secondary schools. Finally, the patients added two items, one on the accessibility of the service and one on the atmosphere in the waiting room.
Refinement of questionnaire (phase 3). We performed two postal pilot studies. In the first one, 696 consecutive patients or carers were sent questionnaires within 48 hours of their request, stratified for type of contact. No reminders were sent. In total, 285 (41%) questionnaires were returned. After studying the numerous written comments by respondents, we rewrote or replaced questions that were ambiguous, confusing or had a non-response of over 20%. In general, respondents found many of the questions too long or complex. Before further testing took place, the revised questionnaire was presented to 13 patients who had recently contacted a cooperative. Apart from a few rephrasings, one item was added concerning accessibility of the pharmacy. We then decided to perform only a small second pilot without reminders: 180 postal questionnaires were sent, 87 (48%) were returned. This time only four items still had a non-response of over 20%. Since all of these items were considered relevant from previous discussions with patients, they were rephrased rather than removed.
Large-scale evaluation
The final concept questionnaires consisted of five sections: general background, telephone nurse, doctor, organization and follow-up/miscellaneous. The three mid-sections consisted of multiple items using 10-point response scales (1–10) plus the option ‘not applicable’. The total number of items varied per type of contact; telephone advice, centre consultation and home visit had 14, 29 and 23 items, respectively.
All GP cooperatives in The Netherlands were invited to participate in the study through widespread advertisements in a national medical paper.
Between March 2003 and June 2004 this resulted in the participation of 26 GP cooperatives, serving around a quarter of the total Dutch population. Two GP cooperatives were excluded due to logistical problems. All GP cooperatives sent postal questionnaires to 200 consecutive patients in all three contact strata within 48 hours of contact and a reminder after 10 days. Patients who had died were excluded from the mailing list. Questionnaires were received by the authors and entered in a database.
In one GP cooperative, a test of test–retest reliability was performed among all respondents. In examining the reproducibility of a measure, the time interval must be sufficiently short to assume that the underlying process is unlikely to have changed.18 Therefore, it was decided to send the same questionnaire to respondents within a week after their first response.
In three of the participating GP cooperatives an analysis to compare respondents with non-respondents was performed using baseline data on sex, age, type of insurance, trauma, part of the day and reason for consultation, as coded in the International Classification of Primary Care.19 A further analysis was performed in five other, also randomly chosen GP cooperatives to study more personal reasons for non-response. At the bottom of the reminder letter a strip had been attached that could be filled out, teared off and returned through an enclosed return envelope. Patients who would not return a questionnaire were asked to tick one of four pre-structured reasons for non-participation: forgotten/not interested, too ill, dissatisfied, language problem or to add an own comment.
Statistical analysis. Principal components analysis (PCA) with varimax rotation was used to check the structure that was assumed in the developmental phases of the questionnaire. Reliability of the scales was expressed using Cronbach's alpha coefficients. Corrected item-total correlations were calculated within all scales. As a large proportion of the respondents had at least one missing (or ‘not applicable’) answer, imputation techniques were used prior to the analyses to keep the variance and covariance unaffected (expectation maximization).20
The test–retest reliability was assessed by calculating the intraclass correlation coefficient (ICC). In general, an ICC of 
0.70 provides confidence in retest reliability.18 The paired Student's t-test was used to study differences between the first (T1) and second responses (T2).
The extent to which items and scales discriminated between GP cooperatives was expressed with the F-statistic, resulting from the one-way analysis of variance.
The non-response analysis was performed using the chi-square test.
SPSS 11.5 was used for all statistical analyses.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Patient characteristics
Twenty-four GP cooperatives participated in the study, receiving a total of 14 400 postal questionnaires for the three types of contact. In total, 7520 questionnaires were returned (52.2%): 2352 for telephone advice (49.0%), 2512 for centre consultation (52.3%) and 2656 for home visit (55.3%). Patient characteristics are presented in Table 1.
|
Non-response
From three GP cooperatives, in total 1636 of 1800 patients who had received a postal questionnaire were retrieved from the electronic medical records (9% missing cases), and divided into a response group (n = 828, 51%) and a non-response group (n = 808, 49%). A higher response was found among men (P = 0.042), age groups between 5 and 14 and between 45 and 74 (P < 0.001), and privately insured (P = 0.001). No differences in response were found for type of contact, trauma, reason for consultation and part of the day (data not shown). Neither sex nor type of insurance was found to have an effect on satisfaction scores. The relation between age and satisfaction was less clear, since both higher and lower levels of satisfaction seemed to be overrepresented, but showed little, if any, overall impact.
In five other cooperatives (3000 questionnaires sent), a total of 463 reminder strips were returned by patients who did not fill out a questionnaire, representing a mean feedback of 15.4% for all types of contact. The main reasons for non-response were ‘forgotten/not interested’ (n = 160, 34.6%) and ‘too ill’ (n = 83, 17.9%). Only 30 patients (6.5%) stated dissatisfaction as reason for non-response.
Finally, we analyzed whether the response rate of a participating GP cooperative was related to satisfaction scores. Response rates ranged from 36 to 57% for telephone consultation (mean 49%, SD 5.6), from 39 to 67% for centre consultation (mean 52%, SD 7.6) and from 41 to 74% for home visit (mean 55%, SD 7.7), but we found no relation between the response rate per GP cooperative and any of the scales for any type of contact [n = 24; Pearson (2-tailed) not significant].
Reliability
Scales and items.
PCA clearly confirmed the three-component structure that was developed prior to the first pilot study, explaining 77, 72 and 83% of the total variance within telephone advice, centre consultation and home visit, respectively. Corrected item-total correlations were all (very) high, apart from the organization scale in the questionnaire on telephone contact (Tables 2
–4). Cronbach's alpha scores exceeded 0.70 for all scales (Table 5).
|
|
|
|
), mean score on T1 and T2 and paired t-test, intraclass correlation coefficient (ICC) with numbers of patients in the retest reliability, and comparison between GP cooperatives using the F-valueFor all types of contact, interscale correlations were fairly high, ranging from 0.59 to 0.69 for telephone nurse and doctor, from 0.52 to 0.71 for telephone nurse and organization, and from 0.53 to 0.56 for doctor and organization.
Test–retest reliability. Of all 600 questionnaires that had been sent, 338 were returned (57%). All 338 respondents received a second questionnaire (retest), 155 of which were returned (45%).
Analysis of the retest data shows that the differences in satisfaction between T1 and T2 are small (Table 5). A decrease in satisfaction appeared to be significant three times (centre consultation: telephone nurse, overall judgement; home visit: doctor) and marginally significant once (centre consultation: doctor). The results for organization show no significant differences for any of the three contact forms. The ICCs range from 0.787 (telephone advice, nurse) to 0.951 (home visit, doctor), which are all very satisfactory.
Discrimination between GP cooperatives
No items in the doctor scale discriminated between GP cooperatives (Tables 2–5). The only significant item in the telephone nurse scale turned out to be ‘taking time to talk’ (P = 0.043 for telephone advice; not significant for other forms of contact). This contrasted with the organization scale in which almost all items discriminated between the GP cooperatives in all types of contact.
Patient evaluation
In general, respondents were very satisfied. Combining all forms of contact, overall satisfaction scores ranged from 7.6 to 8.0 for the telephone nurse, from 7.9 to 8.3 for the doctor and from 7.4 to 7.8 for the organization (on a scale 1–10).
Respondents who only received telephone advice gave lower overall scores on all scales than respondents who received other forms of contact (P < 0.001), while respondents receiving a centre consultation scored lower than those who were visited by the doctor (P < 0.030 or less for all differences). On the question ‘did you receive the care that you hoped or?’ (section five, follow-up), respondents answered ‘no’ in 21.1% of telephone advice, 12.1% of centre consultation and 8.8% of home visit cases (P < 0.001 for all differences).
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
These findings indicate that all three questionnaires have a satisfactory reliability and seem suitable for a broad range of patients contacting out-of-hours GP cooperatives.
Content validity of the questionnaires appears to be ensured by the combination of literature research and exchange with both patients and health care professionals. Construct validity of the scales was supported by the PCA as well as the high corrected item-total correlations within the scales. The questionnaires have a satisfactory internal consistency, with Cronbach's alpha coefficients exceeding 0.70 for all scales. Furthermore, the test–retest analysis showed high intraclass correlation coefficients for all scales.
The decrease in satisfaction found in several scales in the retest analysis may indicate that satisfaction is not as stable a quantity as is assumed. Others have also reported a decrease in satisfaction over time.13 Satisfaction with centre consultation appeared to decrease more strongly than with the other two types of contact.
In this study, a high overall non-response rate of 47.8% was encountered. This may in part have been caused by the rather long questionnaires, although Salisbury et al.21 only found small differences in response between long and short questionnaires evaluates out-of-hours primary health care. Our non-response analysis was performed in 3 GP cooperatives only (12.5%), yet the variables that differed significantly between the response and the non-response groups (sex, age, type of insurance) did not appear to have any effect on the satisfaction scores. Overall, 6.5% (30 out of 463) of the patients who did not fill out a questionnaire but who did send a reason for non-response (through a reminder strip) reported to be dissatisfied. If we would assume that respondents returning a questionnaire were dissatisfied if they had an average score under 6.0 (for which it seems there is broad consensus in The Netherlands), overall 8.0% of the respondents would have been dissatisfied. Therefore, the dissatisfaction rates within the extra 15% of reactions seemed in broad agreement with the 52% response that had already been described. Finally, no relation was found between the response rate of the participating GP cooperatives and any of the mean scale scores, so that overall the results seem generalizable towards all users of the out-of-hours services. However, more thorough research is still needed to confirm this hypothesis.
Although PCA is an exploratory technique, the results confirmed the hypothesized structure of scales and items that was chosen before the start of the field phase (using scales for telephone nurse, doctor and organization). In some scales, the number of items remained higher than necessary. At this stage, we decided not to reduce the number of items to be able to study which questions would discriminate best between GP cooperatives. In the quest of national benchmarking, this could perhaps then serve as an extra criterion in the final reduction of questionnaire items. Unfortunately, only items in the scale for organization showed significant differences that could allow for such an approach. Since the returned questionnaires could not be linked to the individual health carers, it was not possible to determine whether questionnaire items could discriminate between individual nurses or doctors. In addition, item reduction could perhaps focus on the items with the lowest mean (corresponding to aspects of health care that give most room for improvement), while at the same time keeping the Cronbach's alpha acceptably high (e.g. >0.70). In this perspective, the questionnaire on home visits seems least of use, showing highest item means and interscale correlations, while representing only 10–15% of all patient contacts.
Yet another approach would be to reduce the number of items per scale based on new focus group discussions, in an attempt to define the items that are considered most relevant for judging the quality of care.
Despite the relatively recent changes and negative publicity in our country, overall satisfaction with the out-of-hours care by GP cooperatives appears to be high. Respondents who received telephone advice were least satisfied with the telephone nurse. Our findings seem in accordance with other studies, although a difference in satisfaction between centre consultation and home visit was not reported elsewhere.7,8,11,14,16,17,22 Items with the lowest means may lead the way toward quality improvements. For example, within the scale telephone nurse, issues like reassurance and advice should perhaps receive more attention in training programmes. Similarly, factors like accessibility by telephone, general information on the out-of-hours service and further integration of services from the cooperatives and pharmacies deserve extra attention.
In conclusion, we have developed a reliable questionnaire for a broad range of patients in out-of-hours primary health care. However, future research should focus on further item reduction and, ultimately, on the question whether it is possible at all to drive up the standards of care by differentiating satisfaction levels between GP cooperatives.
|
Declaration |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Funding: All but three participating GP cooperatives donated a prefixed fee to cover all expenses of the study and receive an individual feedback report on the evaluation results of their cooperative.
Conflicts of interest: none.
|
Acknowledgments |
|---|
We thank all participating patients, telephone nurses, GPs, staff members of GP cooperatives and colleagues who provided copies of unpublished questionnaires. The authors would also like to express their gratitude to Hanneke Janssens, Marjon Reumkens and Esther van der Plasse for their help with the pilot studies and to Carlien Erlings for assisting on the retest and non-response studies.
|
Notes |
|---|
Moll van Charante E, Giesen P, Mokkink H, Oort F, Grol R, Klazinga N and Bindels P. Patient satisfaction with large-scale out-of-hours primary health care in The Netherlands: development of a postal questionnaire. Family Practice 2006; 23: 437–443.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
1 Hallam L and Cragg D. (1994) Organisation of primary care services outside normal working hours. BMJ 309:1621–1623.
2 Olesen F and Jolleys JV. (1994) Out of hours service: the Danish solution examined. BMJ 309:1624–1626.
3 Salisbury C. (1993) Visiting through the night. BMJ 306:762–764.[ISI][Medline]
4 Pitts J. (1988) Hours of work and fatigue in doctors. J R Coll Gen Pract 38:2–3.[ISI][Medline]
5 Handysides S. (1994) Morale in general practice: is change the problem or the solution. BMJ 308:32–34.
6 Richards T. (1999) Patients' priorities. BMJ 318:277.
7 Hansen BL and Munck A. (1998) Out-of-hours service in Denmark: the effect of a structural change. Br J Gen Pract 48:1497–1499.[ISI][Medline]
8 Christensen MB and Olesen F. (1998) Out of hours service in Denmark: evaluation five years after reform. BMJ 316:1502–1505.
9 Cragg DK, Campbell SM, Roland MO. (1994) Out of hours primary care centres: characteristics of those attending and declining to attend. BMJ 309:1627–1629.
10 Baker R. (1996) Characteristics of practices, general practitioners and patients related to levels of patients' satisfaction with consultations. Br J Gen Pract 46:601–605.[ISI][Medline]
11 Salisbury C. (1997) Postal survey of patients' satisfaction with a general practice out of hours cooperative. BMJ 314:1594–1598.
12 Cragg DK, McKinley RK, Roland MO, et al. (1997) Comparison of out of hours care provided by patients' own general practitioners and commercial deputising services: a randomised controlled trial. I: The process of care. BMJ 314:187–189.
13 McKinley RK, Manku-Scott T, Hastings AM, French DP, Baker R. (1997) Reliability and validity of a new measure of patient satisfaction with out of hours primary medical care in the United Kingdom: development of a patient questionnaire. BMJ 314:193–198.
14 Shipman C, Payne F, Hooper R, Dale J. (2000) Patient satisfaction with out-of-hours services; how do GP co-operatives compare with deputizing and practice-based arrangements? J Public Health Med 22:149–154.
15 McKinley RK, Stevenson K, Adams S, Manku-Scott TK. (2002) Meeting patient expectations of care: the major determinant of satisfaction with out-of-hours primary medical care? Fam Pract 19:333–338.
16 Pickin DM, O'Cathain A, Fall M, Morgan AB, Howe A, Nicholl JP. (2004) The impact of a general practice co-operative on accident and emergency services, patient satisfaction and GP satisfaction. Fam Pract 21:180–182.
17 van Uden CJ, Ament AJ, Hobma SO, Zwietering PJ, Crebolder HF. (2005) Patient satisfaction with out-of-hours primary care in the Netherlands. BMC Health Serv Res 5:6.[CrossRef][Medline]
18 Streiner DL and Norman GR. (2003) Health Measurement Scales: A Practical Guide to Their Development and Use (3rd edn) (Oxford University Press, Oxford).
19 ICPC-2. (1998) International Classification of Primary Care. (Oxford University Press, Oxford).
20 Dempster AP, Laird NM, Rubin DB. (1997) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc 39:1–38 Series B.
21 Salisbury C, Burgess A, Lattimer V, et al. (2005) Developing a standard short questionnaire for the assessment of patient satisfaction with out-of-hours primary care. Fam Pract 22:560–569.
22 Leibowitz R, Day S, Dunt D. (2003) A systematic review of the effect of different models of after-hours primary medical care services on clinical outcome, medical workload, and patient and GP satisfaction. Fam Pract 20:311–317.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||