Family Practice Advance Access originally published online on June 24, 2007
Family Practice 2007 24(4):336-342; doi:10.1093/fampra/cmm025
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
© 2007 The Author(s)
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/2.0/uk/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
The GPs' Experiences Questionnaire (GPEQ): Reliability and validity following a national survey to assess GPs' views of district psychiatric services
a Norwegian Knowledge Centre for the Health Services
b University of Oslo
c University of Bergen, Norway
Correspondence to: Oyvind A Bjertnaes, Norwegian Knowledge Centre for the Health Services, PO Box 7004, St Olavs plass, 0130 Oslo, Norway; Email: oyvind.andresen.bjertnes{at}kunnskapssenteret.no
Received 2 January 2007; Revised 16 April 2007; Accepted 30 April 2007.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
Background. The measurement of patient and professional views of quality are important components in the evaluation of health care delivery.
Objective. To describe the development and evaluation of the GPs' Experiences Questionnaire (GPEQ) for assessing the quality of community mental health clinics in Norway.
Methods. Design: Literature review, GP interviews, pre-testing of questionnaire items and a cross-sectional national survey. Setting: Postal survey of GPs in Norway evaluating 73 community mental health clinics in the five health regions in Norway during spring of 2006. Subjects: Three thousand four hundred and sixty-three GPs were sent a postal questionnaire with the GPEQ and were asked to assess their community mental health clinic responsible for general adult psychiatric services.
Results. Two thousand one hundred and thirty (61.5%) GPs returned a completed questionnaire. Low levels of missing data suggest that the questionnaire is acceptable. Factor analysis identified five scales: workforce situation (four items), discharge letter (three items), competence (four items), guidance (three items) and emergency situations (two items). All scales met the criterion of 0.7 for Cronbach's alpha and test–retest correlations were 0.72–0.87. The results of validity testing were as hypothesized with scale scores significantly related to knowledge of the community mental health clinic, overall satisfaction, negative experiences with the clinic, waiting time and acceptance of referrals.
Conclusions. The GPEQ is a self-administered questionnaire that includes the most important aspects of the GPs' experience of quality at community mental health clinics. All scales have good evidence for internal consistency, test–retest reliability and validity.
Keywords. Community mental health centres, consumer satisfaction, quality of health care, reliability and validity, survey.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
In recent years there has been a steady growth in the availability of quality indicators that are designed to inform patients, providers and policy makers about the quality of health care provision.1,2 In addition to traditional clinical measures of outcome, questionnaires are increasingly used to assess the perceptions of health professionals and patients in relation to health care quality.3
The Norwegian Ministry of Health has decided that satisfaction with co-operation between health care providers shall form part of the national quality indicator system.4 The first national evaluation relates to the primary and secondary care interface, specifically GP experiences with community mental health clinics. The clinics are professionally independent units responsible for a substantial part of general adult psychiatric services in specific geographic areas. Community mental health clinics are part of secondary health care and are a key component in the national plan for improving the quality of psychiatric services in Norway. GPs refer patients to the clinics, maintain contact with patients while they are treated by the clinics, and have a central role after patients have been discharged from the clinics.
In the UK and other countries, GPs have been perceived as an important component of service evaluation and for the identification of priorities for change. District health authorities have used GP evaluations to gain a broader view of services provided by the district.5–8 GPs were asked using postal questionnaires about quantity and quality of a range of health services in primary or secondary care. Many services received positive assessments from the GPs, but psychiatric services were among the poorest rated services on quality.5–7 Other studies have included GP evaluations of specific forms of care including public health nursing, a domiciliary hospice service, an accident and emergency department and clinical allied health outpatient services at a hospital.9–12 Most studies have used a survey approach and while questionnaires have differed most have assessed GPs' views of quality, accessibility and satisfaction.
GP evaluation of psychiatric services includes studies relating to specific services within primary care,13,14 GP evaluation of services in secondary care15–18 and collaboration and satisfaction with mental health professionals in general.19 The objectives of these studies have differed, but the importance of good co-operation between GPs and mental health professionals has been the main impetus for such evaluation. The aspects of care evaluated by GPs have also varied, but satisfaction with one or several aspects of other mental health services is a common element. All studies used some form of survey approach. For the most part, GP evaluations of mental health services have been quite negative, and several improvement areas were identified by all studies.
These studies informed the development of the questionnaire and data collection methods within the present study. However, none of the studies were based on a validated measurement instrument.20 Therefore, we conducted an extensive development and validation project, aimed at securing the reliability and validity of the questionnaire that was used in a national survey. This study describes the development and evaluation of the GP experiences questionnaire (GPEQ) for assessing the quality of 73 community mental health clinics in Norway.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
Development of the questionnaire
The development of the questionnaire followed the identification of domains and items of potential relevance to GPs' experience with quality of community mental health clinics. We searched Medline for articles relating to GPs' evaluation of quality at the clinics including accessibility, competence, discharge information, exchange of information, practice relevant education, professional guidance and waiting time.13–19 Cost considerations meant that articles that were retrieved were Anglo-American or Scandinavian. However, foreign language articles with an English abstract were considered and an article in German that was relevant was translated.17
Qualitative interviews were undertaken with 26 GPs to assess important factors associated with GP experiences with community mental health clinics. We interviewed GPs both in cities and small municipalities in different parts of Norway. As part of a reference group, six GPs and five clinic staff at community mental health clinics revised the initial list of domains and items considered to have relevance in this setting. The questionnaire was piloted among GPs in two Norwegian counties in the autumn of 2005. Further changes were made following the analysis of the pilot material.20
The process of questionnaire development was designed to ensure content validity, that is, the extent to which the items adequately address important aspects of GP experiences. The 22 items use a five-point response format from ‘not at all’ to ‘very large extent’ and are summarized in Table 1. The GPs were asked to evaluate the community mental health clinic with responsibility for general adult psychiatric services in their locality. The questionnaire instructed the GPs to base the evaluation on their own experiences with the clinic, and normative criteria were not presented as a benchmark against which the quality of services could be compared.
|
The questionnaire also included a further nine questions relating to background and context. It takes approximately 5 minutes to complete the questionnaire.
Data collection
With the exception of GPs who took part in the pilot, the questionnaire was mailed to all Norwegian GPs (n = 3463) in January 2006. Non-respondents were sent three reminders, with 10 days between each contact. GPs received no remuneration for participating in the study.
Statistical analysis
Items were assessed for missing data. Factor analysis was used to assess the underlying structure of the 22 items.21 Items with poor factor loadings were considered for removal from the final questionnaire. The internal consistency reliability of the resulting scales was assessed using item–total correlation and Cronbach's alpha. The former measures the strength of association between an item and the remainder of its scale. It was hoped that the level of correlation would exceed 0.4. The latter assesses the overall correlation between items within a scale. For a scale to be considered sufficiently reliable, an alpha value of 0.7 has been recommended.22,23
Test–retest reliability was assessed by sending a second questionnaire to a sample of 270 GPs approximately 10 days after they returned the first questionnaire. The level of agreement between the two sets of scores was assessed using the intra-class correlation coefficient.
Construct validity was assessed through correlations of scale scores and comparisons with GP responses to five additional questions included within the questionnaire. It was hypothesized that scales measuring related aspects of GP experiences would have higher levels of correlation than those measuring unrelated aspects of GP experiences. It was also hypothesized that scale scores would be correlated with global satisfaction and negative episodes relating to the clinic. Further, it was hypothesized that scores would be correlated with perceived waiting time for patients and acceptance of referrals.5,6,17 It was hypothesized that scale scores would be correlated with knowledge of the clinic.14,24
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
Data collection
Of the 3463 GPs mailed a questionnaire, 2130 (61.5%) responded. Respondents and non-respondents were compared in relation to data that were available from the Norwegian Medical Association (NMA) and the Norwegian Labour and Welfare organization: age, gender, number of new patients needed to fill the GP's list, other medical responsibilities/employment, whether the GP was part of a group practice or not, years of service and whether the GP was a member of the NMA or not. With the exception of age, we found statistically significant differences between responders and non-responders on all variables, the largest difference being for other medical responsibilities/employment; 43.6% of respondents had other employment compared to 34.2% of non-respondents (P < 001).
Statistical analysis
The levels of missing data and descriptive statistics for the 22 items are shown in Table 1. Levels of missing data were acceptable for all items, ranging from 0.4% to 8.0%. The four items covering the workforce situation at the community mental health clinics had highest levels of missing data, ranging from 5.6% to 8.0%. Most of the mean item scores were in the middle of the 5-point scale with some skewed towards negative experiences. The lowest and highest mean scores were for the items relating to offer of organized guidance/professional seminars and competence to assess/treat patients, respectively.
Two of the items relating to waiting time and acceptance of referrals were removed because they failed to have adequate loadings within important factors. Table 2 shows that factor analysis without these items produced five factors, which accounted for 70.0% of the total variation. Factor loadings ranged from 0.30 to 0.89. Three of the items had loadings above 0.3 on two factors, and were therefore considered for removal from the instrument.
|
Of the remaining 20 items, four performed poorly in the results of tests of internal consistency: necessary feedback during treatment, information about the treatment at the clinic, contact in situations where the GP needs help and if the clinic takes GPs' opinions of the patients situation seriously. Two of these items also performed poorly in the factor analysis, loading onto two factors. These four items were removed from the questionnaire.
The final 16 questions contribute to five scales, which can be described as emergency situations, discharge letter, competence, guidance and workforce situation. The levels of item–total correlation for the five scales are all acceptable and range from 0.59 to 0.77 (Table 1). The alpha values meet the criterion of 0.7, ranging from 0.77 to 0.89 for guidance and workforce situation, respectively (Table 1).
The scale scores are approximately normally distributed with means between 44.77 and 54.87 on the 0–100 scale where higher scores represent better experiences. The exception was the scale of guidance which had a distribution skewed towards poorer experiences and a lower scale score of 30.95.
Of the 270 GPs mailed a test–retest questionnaire, 162 (60.0%) responded. The test–retest correlations are shown in Table 1. All five scales produced acceptable reliability estimates ranging from 0.72 for discharge information to 0.87 for guidance.
Table 3 shows the results of validity testing which are all significant. The correlations between the five scales range from 0.34 to 0.62, the largest being for the scales of competence and discharge letter and the lowest being for the scales of guidance and workforce situation. The competence scale has moderate to large correlations with all other scales, ranging from 0.52 (guidance) to 0.62 (discharge letter).
|
All five scales have moderate to large correlations with overall satisfaction, ranging from 0.45 to 0.64 for the guidance and competence scales, respectively. Negative experiences with the clinics have small to moderate negative correlations with the scales, ranging from 0.24 to 0.37 for the guidance and competence scales, respectively. Knowledge of the clinic has small negative correlations with the scales, ranging from 0.07 to 0.24 for the workforce situation and guidance scales, respectively. Finally, perceived waiting time and acceptance of referrals have small to moderate negative correlations with all scales, the largest correlations being for the competence scale.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
The GPEQ is a short self-completed questionnaire that is acceptable to GPs while maintaining comprehensibility in its coverage of important domains of GPs' experience with the quality at the community mental health clinics. Questionnaire development was based on literature review and the views of GPs and clinicians from the clinics. The GPEQ has undergone a rigorous process of testing for reliability and validity, which supports its application as a measure of GP experiences of quality at the community mental health clinics. The response rate of 61.5% is acceptable, but we found some differences between respondents and non-respondents. However, GPEQ scores have a limited association with these variables (results not reported) and they are not expected to affect the results of validity and reliability testing.
The results are presented as five scales scored from 0 to 100. The five scales comprise 16 items relating to emergency situations, discharge letter, competence, guidance and workforce situation. The content of the five scales are underpinned by a literature review and qualitative interviews with GPs. The results of the factor analysis support the existence of the five scales. Following the removal of 6 of the 22 items, the five scales were found to have good levels of internal consistency reliability and produced test–retest estimates in excess of 0.7. The six poorly performing items were not found to make an adequate contribution to any of the five scales, their removal serving to improve internal consistency reliability.
GPEQ scale scores were approximately normally distributed with mean scores around the midpoint of the score range which shows that they have the potential to measure cross-sectional and longitudinal differences in GP experiences of the clinics. The five scales have good evidence of construct validity. The significant relationships between the scale scores and knowledge of the clinic, overall satisfaction, negative experiences, waiting time and acceptance of referrals followed the hypotheses. The largest correlations were found between overall satisfaction and the scale scores that lends support to the approach of asking concrete questions about different aspects of quality which gives the clinics more useful information for quality improvement work. The GPs' assessment of competence at the clinics had the largest level of correlation with the other scales and GP responses to questions relating to general satisfaction, negative experiences, perceived waiting time and acceptance of referrals. This highlights the central importance GPs give to competence in their evaluation of the quality of the community mental health clinics.
Two items that were used for validation purposes, perceived waiting time for patients and acceptance of referrals, had significant correlations with the five scales. Both items deal with accessibility to the clinics, an important consideration for patients and GPs. Hence, these items can possibly supplement the five scales as further measures of quality at the clinics. Both items could be part of the questionnaire if accessibility is an important component of the evaluation.
As in other studies,13–19 GPs in this study are quite critical to the community mental health clinics, especially regarding guidance from the clinics. Our study reports GP evaluation on dimensions of importance to GPs, but it is obvious that some dimensions are more important than others; for instance, competence is highly prioritized compared to other dimensions.17,18 However, it has been found that practice relevant continuing education in psychiatry is highly valued by GPs,17 suggesting that community mental health clinics must not disregard the poor scores on this dimension.
Scores on the five scales have been presented for each of the community mental health clinics in the national quality indicator system.4 GPs' views supplement standard user surveys in the national system, and are presented together with other quality measures for institutions within psychiatric secondary care. The association with costs and other quality indicators, including surveys of users, will be assessed in future research.
The construction of community mental health clinics in Norway is a central part of initiatives designed to improve services for psychiatric patients. The Norwegian Research Council invests a substantial amount of resources to evaluate the development of the services. It is important to base such evaluations on reliable and valid instruments and the GPEQ is now available for assessing GP evaluations of the quality at community mental health clinics. Within Norway, each community mental health clinic now has the opportunity to discuss priorities for improvement with local GPs and scores produced by the five GPEQ scales will inform such a process.
In summary, the GPEQ is acceptable to GPs and has good evidence for data quality, internal consistency, test–retest reliability and validity. The GPEQ is being used to measure GP experiences with such clinics throughout Norway, and will be used in future surveys as part of the national quality indicator system. The questionnaire is recommended for future applications designed to assess GP experiences of quality at the community mental health clinics in Norway. It is also recommended that the GPEQ be considered by other Scandinavian and European countries as a means for assessing the quality of community mental health clinics following consideration of organizational aspects of care and necessary cross-cultural validation work.
|
Key points |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
- The measurement of GPs' experiences with secondary care is an important component in the evaluation of health care delivery.
- The GPs' Experiences Questionnaire (GPEQ) is based on a review of the literature and the views of GPs and health professionals.
- The GPEQ has good evidence for data quality, reliability and validity.
- The GPEQ is recommended as a measure of GP experiences with community mental health clinics in Norway.
|
Declaration |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
Source of funding: Norwegian Knowledge Centre for the Health Services.
Conflicts of interest: None.
|
Notes |
|---|
Bjertnaes OA, Garratt A and Nessa J. The GPs’ Experiences Questionnaire (GPEQ): reliability and validity following a national survey to assess GPs' views of district psychiatric services. Family Practice 2007; 24: 336–342.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionKey pointsDeclarationReferences |
|---|
1 Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med (1996) 335:966–970.
2 Marshall MN, Shekelle PG, McGlynn EA, Campbell S, Brook RH, Roland MO. Can health care quality indicators be transferred between countries? Qual Saf Health Care (2003) 12:8–12.
3 McColl E, Jacoby A, Thomas L, et al. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess (2001) 5:1–256.[Medline]
4 Directorate for Health and Social Affairs. Free hospital choice Norway, 2007. http://www.sykehusvalg.no. (accessed on June 5, 2007).
5 Hicks NR, Baker IA. General practitioners' opinions of health services available to their patients. BMJ (1991) 302:991–993.
6 Hull SA, Dennis N, Loughlin L. GP opinions of available health services. BMJ (1991) 302:1405–1406.
7 Brockway CR, Jones KE. Survey of the opinions of general practitioners about health services in a rural setting. Public Health (1993) 107:45–52.[CrossRef][Web of Science][Medline]
8 Horobin JM, Hester SB, Macdonald S, Baijal E. Obtaining the views of general practitioners on the services to which they refer patients—a locality approach. Br J Gen Pract (1997) 47:571–572.[Web of Science][Medline]
9 Hayes CH, Hynes M, O'Herlihy B. Public health nursing in Ireland: the general practitioners' view. Ir J Med Sci (1992) 161:105–107.[Web of Science][Medline]
10 Seamark DA, Thorne CP, Jones RV, Gray DJ, Searle JF. Knowledge and perceptions of a domiciliary hospice service among general practitioners and community nurses. Br J Gen Pract (1993) 43:57–59.[Web of Science][Medline]
11 Choyce MQ, Maitra AK. Satisfaction with the accident and emergency department—a postal survey of general practitioners' views. J Accid Emerg Med (1996) 13:280–282.
12 Grimmer K, Bowman P. A survey of general practitioners: implications for the hospital/community interface. J Qual Clin Pract (1996) 16:185–194.[Medline]
13 Stansfeld SA, Leek CA, Travers W, Turner T. Attitudes to community psychiatry among urban and rural general practitioners. Br J Gen Pract (1992) 42:322–325.[Web of Science][Medline]
14 Thomas RV, Corney RH. Working with community mental health professionals: a survey among general practitioners. Br J Gen Pract (1993) 43:417–421.[Web of Science][Medline]
15 Bindman J, Johnson S, Wright S, et al. Integration between primary and secondary services in the care of severly mentally ill: patients' and general practitioners' views. Br J Psychiatry (1997) 171:169–174.
16 Kettles AM, Chisholm A. General practitioners' views on a mental health service. J Nurs Manag (1998) 6:37–43.[CrossRef][Medline]
17 Spiessl H, Semsch I, Cording C, Klein HE. Interviews with physicians in private practice as a starting point for quality improvement in hospitals. Z Arztl Fortbild Qualitatssich (2001) 95:419–423.[Medline]
18 Heskestad S, Horneland M. Primary care physician's evaluation of psychiatric outpatient clinics. Tidsskr Nor Laegeforen (1994) 114:2402–2404.[Medline]
19 Younes N, Gasquet I, Gaudebout P, et al. General practitioners' opinions on their practice in mental health and their collaboration with mental health professionals. BMC Fam Pract (2005) 6:18.[CrossRef][Medline]
20 Holte TO, Groven G, Iversen H, Bjertnæs ØA, Garratt A. General practitioner's assessment of quality at the community mental health clinics: development of questionnaire and data collection methods. Report no. 7-2006 (2006) Oslo: Norwegian Knowledge Centre for the Health Services.
21 Hair J, Anderson RE, Tatham RL, Black WC. Factor analysis. In: Multivariate Data Analysis with Readings (1995) Englewood Cliffs, NJ: Prentice-Hall International Inc. 364–406.
22 Nunnally JC, Bernstein IH. Psychometric Theory (1994) 3rd ed. New York: McGraw-Hill.
23 Kline P. A Handbook of Test Construction (1986) London: Methuen.
24 Kværner KJ. Nursing home services and the general practitioner. Tidsskr Nor Laegeforen (2005) 125:1016–1017.[Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  O. A Bjertnaes, A. Garratt, H. Iversen, and T. Ruud The association between GP and patient ratings of quality of care at outpatient clinics Fam. Pract., October 1, 2009; 26(5): 384 - 390. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 O. A. Bjertnaes, A. Garratt, and T. Ruud Family Physicians' Experiences With Community Mental Health Centers: A Multilevel Analysis Psychiatr Serv, August 1, 2008; 59(8): 864 - 870. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 O. A. Bjertnaes, A. Garratt, and G. Botten Nonresponse Bias and Cost-Effectiveness in a Norwegian Survey of Family Physicians Eval Health Prof, March 1, 2008; 31(1): 65 - 80. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||