Family Practice Vol. 20, No. 5, 501-502
© Oxford University Press 2003
Editorial |
Family Practice—what makes a good review?
Guy’s King’s & St Thomas’ School of Medicine, London, UK; E-mail: roger.jones{at}kcl.ac.uk
Jones R. Family Practice—what makes a good review? Family Practice 2003; 20: 501–502.
Background
Reviews and reviewers are an essential part of the publishing process. Despite well-described drawbacks, peer review remains a cornerstone of the evaluation, revision and selection of papers for publication. Reviewers’ comments are an important part of editorial decision making.
Reviews are of widely variable quality and utility. This variation has been subject to a good deal of research, and some of the characteristics of good reviewers are beginning to emerge, as well as strategies, particularly feedback and training, which are successful in improving the quality and usefulness of peer reviews. One important factor is, of course, the clarity of instructions given by editors and their journals to reviewers. Many journals have approached this by using structured pro-formas, and encouraging on-line reviewing. My own experience of this is that the process can feel restrictive and reductionist, and tends to encourage convergent and introspective thinking about the paper under review.
At the same time that these concerns about quality, consistency and utility are being discussed, there is an increasing focus on probity and the conduct, reporting and review of research. This is reflected internationally in the USA’s Council on Research Integrity, the Committee on Publication Ethics in the UK and the layers of bureaucracy surrounding the National Health Service’s research governance framework. Journals should now insist on evidence of ethical committee or Institutional Review Board approval of research studies, together with an indication of the funding sources for the study and of competing and conflicting interests of authors. These and other criteria of probity apply with equal force to reviewers, authors and editors.
The editorial process
Family Practice receives at least three times as many papers as it is able to publish, with submission rates and the journal’s impact factor both rising, evidencing increases in quality and volume. Our instructions to authors are being updated to accommodate some of the more recent requirements of research governance and ethics. Authors need to read these instructions, and are well advised to read the journal as well. Traditional paper-based libraries have great advantages over electronic publication and databases—browsing and flicking through paper journals is much easier and more revealing.
The first stage in the publication process is editorial screening, carried out in the context of the journal’s publication policy and its instructions to authors. A small proportion of papers have simply been submitted to the wrong journal and, almost irrespective of quality, would be better placed elsewhere. A similar small number are of such evidently poor quality that it would be inappropriate to use reviewers’ valuable time to evaluate them. A further group consists of papers in the wrong format—unstructured reviews, unsystematic opinion pieces and the like, which do not merit peer review.
Mercifully, a substantial number of papers are those that the editor is delighted to receive, and in these cases the peer reviewer is required to provide expert advice about the quality of the content, rather than the interest or appropriateness of the subject, or its editorial fit, already considered by the editor. Other papers, which present a prima facie case for publication, require review in relation to specific academic or disciplinary content or to a particular methodology such as quantitative statistics or the analysis of qualitative material.
It is usual for two experts in the field to be chosen as reviewers, with the option to seek further advice when, for example, there is dissonance between expert reviewers’ opinions, specific questions about methodological issues or when authors think that they have been misunderstood or misjudged.
Good and bad reviews
Whilst it should be clear by now that peer reviews are essential for editors to obtain advice and guidance, it also follows that the function of reviews may be different under different circumstances; for example, questions about novelty, international relevance and appeal and generalizability may be much better judged by experts in the field than by individual editors.
One characteristic of any review places it firmly in the ‘bad’ category, and that is lateness. Just as authors have a responsibility to conform to journals’ requirements, style and focus, so journals and their editors have a reciprocal responsibility, almost amounting to a duty of care, to their authors. After all, as someone once remarked, all editors are failed authors but, then again, so are most authors. In practice, this means that reviewers should do their utmost to turn the review around within the 2–3 weeks required by the journal and, if for some reason they are not able to do this, to let the journal know as quickly as possible, preferably suggesting an alternative reviewer. Recognizing the pressure on many individuals, editors, this one included, may suggest to senior reviewers that, whilst sticking to rules of confidentiality, they should feel able to request reviews from less hard-pressed members of their research group or department. Most authors’ queries relate to the slow processing of manuscripts which, in turn, is generally related to tardy reviewers. Keeping a database of reviewers should guard against future editors’ disappointment.
The first prerequisite for a good review is that it is timely. The next is that it should provide usable advice to the editor, preferably including a recommendation for or against publication, backed up by a clearly set out series of comments, identifying the strengths and weaknesses of the paper, in a form which can be passed on directly to the authors without causing offence and, hopefully, encouraging and facilitating revision of salvageable manuscripts. Peer review is both a formative and a summative process.
Length matters. Very short reviews are often difficult to operationalize and leave the editor feeling that he is short changing his authors. I remember one particularly pithy review which described a paper as ‘vacuous psychobabble’, and little else. It was difficult to pass this on, constructively, to the authors. Some of the more senior figures in the pantheon of peer review fall into the trap of highly critical, synoptic and, ultimately, almost useless negative reviews.
Conversely, an essay on the philosophical issues surrounding the paper can be equally unhelpful. More than two sides of A4 begins to get unmanageable, particularly if re-submission of a revised manuscript is anticipated. It is not fair to give the authors too much to do or to try to re-write their paper completely for them. Even papers accepted by the highest impact journals are susceptible to further criticism and potential revision, not to mention the later discovery of scientific fraud, which can evade detection by the most diligent and experienced reviewer.
Review requests from Family Practice focus on four main issues—the question being asked, the methodology being used to answer it, the results and other data collected in the study and their interpretation, and generalizability and likely interest to an international academic primary care readership. We need to know whether research questions and papers are novel, interesting and engaging, or repetitive, derivative or plain dull. Often only those familiar with a particular field of enquiry can answer that question. Methodological expertise is important, and Family Practice is often challenged to access the wide range of qualitative and quantitative methodological skills required to cover all the material submitted. Data presentation and interpretation are important, and comments highlighting over-interpretation are particularly helpful, when authors go beyond the scope of the enquiry in their discussions and conclusions.
The final criterion mentioned above, that of wide international interest, is particularly problematic. Family Practice receives papers from all over the world, of hugely varying quality, often reflecting the stage of development of the primary care system from which they come. Encouragement is important, but so are standards, and delicate editorial decisions have to be made to reconcile these imperatives. Parochial pieces, irrespective of their quality, which focus on a single health care system and whose lessons are likely to be inapplicable outside the region or country where the study was conducted are frequently turned down on this basis.
Last, but not least, the format of the report is important. Whilst the on-line electronic review system is working well, and offers a convenient way of passing comments on to authors, many reviewers still prefer to use paper, in which case separating the review from the covering letter, anonymizing it and including the Family Practice reference number and title are essential.
Reviews and reviewers are valuable commodities, and I would like to take this opportunity to thank the large number of reviewers who contribute to the continuing success of Family Practice. The journal’s finances do not, at present, allow us to offer a monetary token of thanks, but reviewers should be sure that they have earned the gratitude of the editor, the journal’s staff and all its contributors and readers.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||