Family Practice Vol. 21, No. 1, 81-82
© Oxford University Press 2004, all rights reserved.
Article |
Can you compare competence and performance? A reply to McKinstry et al.
University of Hull, Postgraduate Medical Institute, Division of Psychological and Primary Care Medicine, Cottingham Road, Hull HU6 7RX, UK
E-mail: p.d.campion{at}hull.ac.uk
Received 3 January 2003; Accepted 8 September 2003.
Campion P. Can you compare competence and performance? A reply to McKinstry et al. Family Practice 2004; 21: 81–82.
The MRCGP video assessment of consulting skills was developed to determine whether candidates for membership of the Royal College of General Practitioners possessed those competencies that a group of ‘experts’ (i.e. GPs on the panel of examiners) considered to be necessary for good clinical practice. It grew out of an assessment paradigm that was overtly criterion-referenced and qualitative,1 in which judgements were to be made according to defined ‘performance criteria’, summarized in Box 3 of the paper by McKinstry and colleagues2 (which, however, omits the full definitions3).
I want to suggest that these authors have committed what in 1949 the linguistic philosopher Gilbert Ryle called a ‘category mistake’.4 Ryle applied the term widely, but its relevance may be seen from the following quote:
"The theoretically interesting category mistakes are those made by people who are perfectly competent to apply concepts, at least in the situations with which they are familiar, but are still liable in their abstract thinking to allocate those concepts to logical types to which they do not belong."
The category mistake seen in this paper is to allocate the MRCGP result (a qualitative judgement about competence) to the same category as a score on a measurement scale (a quantitative measure about patient satisfaction or enablement). So their conclusion, that there is no correlation between these two sets of findings, is flawed.
It is flawed in several ways. First, the authors have used a crude aggregation of all the MRCGP performance criteria to construct a rank ordering of the subjects. While this may appear reasonable, it is clearly not how the exam was designed, and its validity is untested. The MRCGP result is a categorical judgement that a particular doctor possesses a set of competencies, as defined by the exam regulations and workbook: it certainly does not claim to rank-order the candidates in the way McKinstry and colleagues have assumed. The award of merit, based upon a different set of criteria from those defining a pass, actually defines a subset of those passing who also possess certain other competencies. Merit is therefore qualitatively different from pass, and can be regarded as ‘icing on the cake’, an extra accolade, rather than a mark of ‘doing passing better’. The only sense in which the MRCGP data are ordinal is that the two categories ‘pass (including merit)’ and ‘fail’ can be ranked one better than the other.
No only is the MRCGP a categorical judgement, it is a judgement about ‘competence’,5 what a person is able to do. Performance, in this case what a doctor actually does in the consulting room, is the product of competence plus many other factors (systemic and individual) which are likely to have significant effects. While it is quite reasonable to seek an association between competence and performance, it is unreasonable to expect it to be strong.
It follows that the power of a study exploring the relationship between competence and patients' satisfaction should be powered to detect even weak associations. The authors suggest that their study is powered to detect a rank correlation coefficient of 
0.46, although their underlying assumptions about the psychometric properties of the MRCGP measure lead me to question this. Other concerns aside, this would lead to a type II error, of missing a real association though lack of power. If the MRCGP data were considered as categorical (fail, and pass including merit), a contingency table with only two cases (‘fail’) in one column would normally be unacceptable.
On the wider issue of ‘so what?’, it is appropriate and necessary to validate the MRCGP video exam, but the primary comparison should be with other measures of competence, such as the OPTION scale for determining the degree of involvement of patients in decision making.6 Such a study is in progress.
 |
References |
|---|
 Top References |
|---|
1 Biggs J. Teaching for Quality Learning at University. Buckingham (UK): Open University Press; 1999: 141–164
2 McKinstry B, Walker J, Blaney D, Heaney D, Begg D. Do patients and expert doctors agree on the assessment of consultation skills? Fam Pract 2004; 21: 75–80.
3 www.rcgp.org.uk/rcgp/exam/videoworkbook/contents.asp (accessed 18th June 2003) Fam Pract 2004; 21: 75–80.
4 Ryle G. The Concept of Mind. London: Penguin Classic; 2000: 19.
5 Rethans J-J, Norcini J J, Baron-Maldonado M et al. The relationship between competence and performance: implications for assessing practice performance. Med Educ 2002; 36: 901–909[CrossRef][Web of Science][Medline]
6 Elwyn G, Edwards A, Wensing M, Hood K, Atwell C, Grol R. Shared decision making: developing the OPTION scale for measuring patient involvement. Qual Safety Health Care 2003; 12: 93–99.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||