RSS feedsPeer review is slow, potentially biased, and does not even prevent fraud reliably. As the open access movement gathers pace and journal publishing tries out new models, peer review is coming under greater scrutiny.
Academics and researchers want faster access to papers and there is growing concern that a huge number of potentially important findings are lost if they fail to make it through the selective peer review system.
A number of experimental launches in 2006 built momentum towards open peer review. The BMJ, PLoS Medicine, Philica, Naboj and Biology Direct have all recently set up open peer review models.
In some cases, this means posting papers directly onto a journal website without peer review or editorial intervention, and an open peer review process then takes place on the website. Others operate some level of editorial checking or peer review prior to posting the paper.
The anonymity of peer reviewers, both pre- and post-publication, is a key issue. The term “open peer review” implies that the process will ultimately be made completely transparent and the reviewers identified at every stage. In practice, though, this has not been possible in some disciplines.
The BMJ initiated open peer review in 1999 by asking reviewers to sign their reviews, but now leaves it up to the reviewer whether to sign or not. Senior editor Trish Groves says the main reason for adopting open peer review is to try and increase confidence in published work.
“It is an art not a science,” she says. “It is not a great system but it is what we have got. People want to know that science is authoritative. There is great store set by peer review, which is a good reason to try and make it as good as possible. The reality is that it is all just a matter of opinion.”
One study by Peter Rothwell and Christopher Martyn has shown that, on the whole, peer review is best at making articles clearer by improving their readability and clarity, and does not necessarily add any other value to a paper.
Groves says the BMJ has not found it very difficult to persuade reviewers to identify themselves, but acknowledges that more specialist journals may experience greater pressure to maintain anonymity
“We are a general medical journal – it is not a small world, it is not just one bunch of researchers and reviewers. Some people say that in their field their career progression depends on the old boy network. If they write a bad review, they can’t get the next grant.”
Groves says she had hoped that the identification of peer reviewers would improve the quality of reviews. “The worry is that you end up with bland reviews [when they are signed]. We have found that isn’t the case, although, to be honest, we had hoped it would show an improvement.”
Groves questions the value of peer review beyond its function as a selection tool. According to Groves, the editor of the BMJ looked up in Medline those papers that had been rejected after peer review – and found them completely unaltered despite the helpful comments.
“We say, please tell us if your paper has been rejected and send us the peer review comments,” Groves explains. “And show how you have responded to these – this helps us in our appraisal.”
Science journal Nature tested open peer review for three months last summer with limited success. Few authors wanted to submit their papers to online open peer review and the number of commentators was low.
According to some observers, the low take-up experienced by the Nature trial was due to its impermanence. Reviewers may have felt their efforts would be wasted as the site was going to be taken down shortly afterwards.
“There weren’t enough papers, it didn’t go on for a long time and there weren’t enough commentators,” says Nature executive editor Linda Miller. “Some fields revolve around computers, others revolve around the bench. Maybe we would have got more thorough comments but people realised that they might be putting their career on the line if they made comments.
“My guess is the lack of good comments was because there was no incentive. It was an early foray into how we could do it. The culture is not ready to reward those who go out of their way to give great reviews. There is a need to review how we hire and fund to include this.”
Miller says she is satisfied with the 10-15% participation rate of authors submitting papers for open peer review. “I would expect the physical sciences to be the ones with the highest uptake as they are used to Arxiv,” she says. “Nature publishes a lot of biology and I thought biologists would not be grabbed by this, so 10-15% pretty good. I am surprised by the breadth of the take-up. It is a very competitive sector.”
Nature intends to introduce a facility for researchers to comment post-publication this year. “We are working out how to do it now,” says Miller. In the meantime it has furthered the open peer review debate through its “peer-to-peer” blog.
PUBLISHER BYPASS
While the BMJ and Nature tweak the current publishing process, others seek to
use the interactive features of the web to bypass traditional publishers
altogether. Philica was set up by two academics in March 2006. Papers may be
published on any subject immediately on the website. At the beginning of 2007
the site had more than 90 papers and observations, and 200 members. It is listed
on the Directory of Open Access Journals (DOAJ) and it has an ISI number.
Nigel Holt, one of the founders of Philica, says: “Many rejected papers are never published. The content is still of interest but the papers are forgotten and the researcher moves away from the experiment.
“The process of discussion is an important thing. I thought it would be interesting to publish the papers with its reviews. We needed to work out a way of assessing if the paper was any good. The work is published, then users of Philica rate it. As with eBay, where you wouldn’t buy from someone with a very low rating, you might take care if a paper had a veneer of sense but its reviews were poor.
“You can publish articles and observations made as an aside to an experiment – the kind of discussion you have over a glass of wine at a conference, where you might say ‘I found that as well.’ Work stems from that kind of conversation.”
Holt believes that the take-up of the Philica site is likely to be slowed by the current government funding method for university research, the research assessment exercise (RAE).
“What universities are interested in is the RAE, so they are less likely to publish on Philica,” he says. “They will hold onto a paper rather than publish openly.”
The government held extensive consultations last year to try and shift the basis for allocating research funds away from purely numeric measures, to more qualitative assessment. But in the Chancellor’s pre-budget speech in December it became clear that this goal had not been reached. The fear is that universities will continue to try to get published first in top-tier high-impact journals at the expense of open access and open peer review.
The American Public Library of Science (PLoS) has joined the peer review experimenters, launching the PLoS One beta version at the very end of 2006. PLoS One is an open access online journal that accepts papers from any discipline in science and medicine and allows users to annotate articles.
The aim of PLoS is to make as much scientific, medical and technical literature available as possible. It has come in for criticism for not being open enough, as the open peer review occurs after publication. Prior to publication, the editorial team decides whether or not to accept papers, based on technical accuracy and sometimes with the help of peer reviewers.
PLoS managing editor Christopher Surridge says: “We define open access as free to read and available under a licence that allows content to be freely used. There is nothing in that about peer review.
“We felt there were a vast number of authors who don’t have a venue for their work. With PLoS One we wanted to use all the functionality of the web to create a place where any paper of value could be published, so no-one need ever say they haven’t got an open access venue for their work.
“At the same time as creating a big volume journal, we wanted to see if peer review could be made much more transparent. Anything that was valuable was taken. It is a different editorial decision about whether to publish. There is a 70-80% acceptance rate, although it is early days.”
PLoS One’s annotations system lets reviewers make notes directly onto the paper.
“We felt that the commentary system had to be intimately involved with the paper, then the paper becomes just the start of the conversation, but we realise it will be slow to be taken up,” says Surridge.
PLoS One plans to introduce a reviews rating facility. Tagging is also in the pipeline.
Surridge defends the decision to use anonymous peer reviewers before publication. “Editors are identified, so the person who has made the decision to publish is identified. We strongly encourage our referees to comment on the paper once it is published. We didn’t want to change things so much that it would put people off; we wanted contributions clearly peer-reviewed, so they were clearly scientific papers.
“After publication you can’t comment unless you identify yourself. Users know who people are. They are not allowed to hide their identity. We came to that conclusion to ensure civilised discussion. We will see whether that inhibits growth. If it is a serious problem, then we may change our minds. We wanted to get a volume of papers to be published and we didn’t want to apply disincentives. Authors have to stand up and say who they are and put their reports on the line; allowing them to be open to attack by people who hide behind anonymity did not seem right.”
PLoS One charges authors for publication but stresses the intention is not to prevent anything being published because the author cannot afford the fees, and waives fees where necessary.
“We believe publication is part of research and ideally within the funding for research there should be an amount for publishing, but we realise that some authors do not have access to those funds,” says Surridge. “The people who make decisions about what is published never know whether or not the author is funded.
“At no point has the level of waivers caused us any problems. In fact, the level of papers that are not funded is lower than we budgeted for. If we have problems with a particular discipline or geographical location, we would look to work with other agencies to find the money.”
Peer review is an essential cog in the machinery of scientific progress. Publishers’ efforts to make this cog run more smoothly clearly need to be oiled by the participation of researchers and reviewers. Coupled with a new willingness to value participation in making scientific research more open, the peer review process looks set for exciting developments in 2007.
All Tags: Features
del.icio.us
Digg this
reddit!
