Peer-to-Peer - Blog Posts with Comments

What Nature Physics wants

mclarke — 2009-11-06T12:50:38+00:00

Peer review is the cornerstone of scientific publishing. But it isn't always clear exactly what Nature Physics expects of its referees. The journal explains in its November Editorial (5, 775; 2009). "Whatever you think about a paper, it is vital to explain to us exactly why you think it. Your colleagues among the other reviewers may disagree with your assessment, and we do not base our decisions on a show of hands. Hence detailed critiques carry more weight in informing our decisions than terse affirmations one way or the other (in most cases we would disregard the latter, regardless of who supplied it). A further point to consider is whether the work presented in a paper is similar to what has been done before — in such a case, please explain exactly what has been done previously and indicate where it was published." The Editorial outlines the journal's peer-review process, what the editors look for in a review, how to write the review, and how the editors make their decisions. "Peer review is essential for maintaining the integrity of the scientific record. It's well worth the effort. And we thank all of you who make it."

Nature Chemistry on improving peer review

mclarke — 2009-10-26T13:32:58+00:00

Perceived lapses in the peer-review process often receive a lot of attention, but the majority of researchers declare themselves satisfied with the system even though they would like to improve it. If it is imperfect or broken, how do we fix it? This question is addressed in the November Editorial of Nature Chemistry ( 1, 585; 2009), in light of some blog commentaries which identified prior publications that had not been referenced in a journal paper.
Open peer-review experiments have generally not been very successful because reviewers are less likely to make forthright comments in an open forum. Double-blind peer review is another option, but one must consider the role of the editor who oversees the process, as well as the difficulties of effectively hiding the identity of authors in smaller fields from other experts — especially when many authors regularly cite and discuss their previous work. The Editorial concludes:
"The Royal Society of Chemistry's Dalton and Faraday discussion meetings provide a unique mix of traditional peer review coupled with both comment (by peers) and responses from the authors, but require members of a particular research community to assemble at a conference. It is in some ways similar to the grant proposal review process at, for example, the US National Institutes of Health. However, such a process is clearly not a viable option for every one of the vast number of papers submitted for publication. The journal Atmospheric Chemistry and Physics uses a system in which, after initial assessment by an associate editor, manuscripts are posted online for comment. After referee reports are received, these are also posted online with the manuscript along with author rebuttals. If eventually accepted, a paper is formally published in the journal, whereas those that are not remain available (and citable) as online 'discussions'. This differs from the preprint servers Nature Precedings and arXiv because there is an initial assessment of the suitability of the work (based on more than just scope).
Perhaps a hybrid system could be the solution. Traditional peer review, and a decision to publish, could be followed by a fixed period in which any interested party could post questions or comments and the authors are given the opportunity to respond — all moderated by an editor — before a final version of the article (including comments and responses) is preserved for the record. This would again require a large change in the habits of the community — authors, reviewers and publishers — and previous experiments with commenting on published papers have been far from conclusive."
Sense about Science peer-review survey 2009.
Nature journals' peer review policy and Editorials on the subject.

Myles Allen on planetary boundaries and peer review

mclarke — 2009-09-23T18:00:45+00:00

This post is by Myles Allen of the University of Oxford:
As a vocal supporter of the traditional system of scientists communicating through peer-reviewed channels ( Nat. Geosci. 1, 209; 2008 and associated debate at Peer-to-Peer), I was hesitant about writing a critical Commentary on the Feature 'A safe operating space for humanity' by Johan Rockström et al in the 24 September issue of Nature (Nature 461, 472-475; 2009) in a non-peer-reviewed forum. The Nature and Nature Reports Climate Change editors had clearly thought through this argument: the Feature was not itself peer-reviewed, so no golden rules would be broken in publishing a series of commentaries alongside it in Nature Reports Climate Change.
The problem is that packing the point into a few hundred words, and slipping into the usual bloggers' trap of feeling you have to shout loudly on the internet or no-one will listen, means that the end result reads very black-and-white. Corresponding with Johan Rockström over the past week, it emerges we agree on far more than the tone of my Commentary, 'Tangible targets are critical' (Nat. Rep. Climate Change doi:10.1038/climate.2009.95), probably implies. I understand that Rockström et al. had not originally intended to make the link between a six-degree climate sensitivity, the two degrees target and 350 p.p.m. a focal point (as I read it) of their Feature. Likewise, they observe, reasonably enough, that limiting cumulative carbon dioxide emissions to one trillion tonnes of carbon is just another way of framing the climate boundary, with (as I acknowledge) remarkably similar implications to 350 p.p.m.
If this had been done the old-fashioned way, Rockström et al. would almost certainly have had to qualify their reliance on a six-degree climate sensitivity in the course of the usual to-and-fro with referees. And I in turn would have toned down a lot of my objections. The end result would undoubtedly have been blander on both sides, but would that really have been much of a loss? Following what was, for me, an experiment, I still feel it is very much an open question whether scientific communication in general benefits from direct publication rather than allowing rough edges to be smoothed off through traditional peer-review.
Myles Allen is at the University of Oxford, Parks Road, Oxford, OX1 3PU, UK.
-----------------------------
Context:
Johan Rockström and his co-authors argue in a Nature Feature that to avoid catastrophic environmental change, humanity must stay within defined 'planetary boundaries' for a range of essential Earth-system processes. If one boundary is transgressed, then safe levels for other processes could also be under serious risk, they caution. Seven expert commentaries respond to this proposal in Nature Reports Climate Change, one of which is by Myles Allen. All these articles can be accessed from this index page.
There is an associated Nature podcast in which Johan Rockström is interviewed and editor Ehsan Masood provides further analysis.
Further reading:
A longer paper upon which the Nature Feature is based, 'Tipping towards the unknown', is available at the Stockholm Resilience Centre.
Nature news special: The road to Copenhagen.

Comment 1 (Myles Allen on planetary boundaries and peer review)

Magnus Westerstrand — 2009-09-24T09:42:23+00:00

Am I missing something, the article is published?

http://www.stockholmresilience.org/download/18.1fe8f33123572b59ab800016602/planetary-boundaries-long-version210909.pdf

Comment 2 (Myles Allen on planetary boundaries and peer review)

Hank Roberts — 2009-10-04T16:56:51+00:00

It appears the age of the peer reviewers is pertinent information and should, now that this has been established, be provided to the readers:

http://www.nature.com/naturejobs/2009/091001/full/nj7264-681a.html

NSMB speaks up for peer-reviewers

mclarke — 2009-09-14T09:31:18+00:00

Manuscript peer reviewing is at the heart of the scientific system, but it seems that these duties are often not properly (if at all) recognized by universities, funding agencies or even the rest of the scientific community. This is the main message of the September Editorial in Nature Structural & Molecular Biology, 'The Unsung Reviewer' (16, 899; 2009) The Editorial notes:
"Scientists wear many hats these days. They prepare and teach classes and sit on various committees. Then there are the multiple activities directly related to their research, including grant writing, mentoring students and postdocs, attending conferences, writing papers and reviewing manuscripts. All these duties can take a considerable amount of time and effort and most are recognized as worthy contributions by funding agencies, universities and research institutions when evaluating a scientist's performance. On the other hand, peer reviewing papers seems to be the Rodney Dangerfield ("I get no respect!") of a scientist's duties."
A new report by the Science for Policy project of the US Bipartisan Policy Center notes that peer-review is essential for the scientific system. Journals such as NSMB (and the other Nature journals) would not agree from their own perspective with one of the report's conclusions, that "peer-review is no longer assumed to be a professional obligation". As the Editorial points out, our journals have very broad reviewer pools (running into many tens of thousands for the heavily oversubscribed weekly title Nature, for example), and the editors know and very much appreciate the work that these reviewers put into improving submitted manuscripts.
The SPP report also identifies a need to increase the number of scientists who participate in peer review for federal agencies, making suggestions such as listing such service on grant applications or even making it a requirement for funding. In addition, it urges journals to run a quality system, for example by providing the peer-reviewers with feedback (which the Nature journals do, and which is appreciated by our reviewers - for one reason, because a reviewer can see his or her report in the context of those of the other reviwers of the manuscript). The NSMB Editorial concludes:
"Peer review is often compared to jury duty, a chore that one has to do once in a while as a service to the community. But reviewing manuscripts can be enjoyable and useful at any career stage. True, it takes time, but as one reviewer aptly put it: "I still learn do's and don'ts from reviewing papers, as well as just getting useful information, so it is time spent pretty well." It is also true that one does not get much open recognition from it, and there are limitations on what the journals can do without compromising the reviewers' anonymity. We have in the past publicly acknowledged our reviewers at the end of the year, and will do so again this year. The SPP report stresses that it is important to stress that "peer reviewing manuscripts should be an expected and appreciated aspect of a scientist's career." Of course, we do have a vested interest in this, but we firmly believe that it's important to cultivate a vibrant 'reviewer culture'."

Nature journals' peer-review policies and guidelines.
More on peer review from the Nature journals.

Becoming a peer-reviewer for a journal

mclarke — 2009-09-07T09:41:56+00:00

We received a question at the Nature Network 'Ask the Editor' forum which I thought readers of this blog might find useful.

Q: I once reviewed several manuscripts for a good journal, but it is on behalf of my Ph.D boss. Now, I am a postdoc and wonder how to become a reviewer? Thanks for any suggestions.

A. At Nature, we ask our peer-reviewers to identify anyone who helps them with their review. (The reviewers promise to keep the ms confidential, in advance of being sent it, and undertake to ensure that anyone they show it to also keeps it confidential). Therefore, we often discover (and regularly use) good new reviewers by this method. Many senior reviewers ask junior colleagues to review a ms as part of their mentoring, and are very good at assigning credit to these junior colleagues.
I suggest that if you help your boss or another colleague with his or her reviews in future, you ask that person to name you as a collaborator when he/she sends the journal the review, so your name gets known.
If you have been a co-author on your boss’s papers, you could contact the journal that published the work and offer to review (providing the information that you’ve published in that journal).
I think that as you publish in your own right, journal editors will get to know of you and start to ask you to review for them.
If you meet any journal editors at conferences you attend, you could let them know that you’d be interested in peer-reviewing for their journal.
Good luck!

International peer review improved Irish research rankings

mclarke — 2009-09-03T10:24:47+00:00

This is the text of a Correspondence published in Nature (460, 949; 2009) by Conor O'Carroll of the Irish Universities Association:

Your News story 'Italy outsources peer review to NIH' (Nature 459, 900; 2009) highlights a problem common to many countries with a small population of research scientists. Ireland can be held up as a successful model in addressing this problem because, over the past eight years, funding agencies have moved to fully international peer review.
A few years ago, important research and development ventures were set up with a new infrastructure to attract talented people from abroad. The use of only Irish peer reviewers to allocate millions of research euros to a small number of universities could not stand up to the principles of objectivity, transparency and perceived fairness and would have led to conflicts of interest. Despite initial opposition, exclusively international review is now accepted; researchers want to be benchmarked internationally as well as nationally.
The typical process for research evaluation in Ireland is to consult four or five reviewers by mail for each proposal. Proposals are then assessed by a panel of invited experts, who meet in Ireland. Reviewers may be sourced through international funding agencies, or by letting applicants nominate experts themselves.
Some Italian scientists in your News story express reservations. They may well have a point, as US reviewers will probably not have any detailed knowledge of how research is conducted in Italy. One approach is to have nationals involved, either as observers or in a formal non-voting role. For example, the Irish Health Research Board organizes international mail reviews and panels, but the chair of each is Irish. They cannot participate in selection, but ensure that the correct procedures are followed and can explain the national research-funding policy. International panel members appreciate this local input, which helps them think outside their own national funding system.
Reviewing criteria often include the quality of the project, the researchers and their institutions, and the social and economic impact of the research. It is important that international reviewers focus on the quality of the first two, as the standing of institutions and the probable impact of a project can be harder for them to evaluate. Also, they should not get involved in detailed budgetary considerations, as these are strictly national.
Things have changed radically in Ireland's research over the past ten years. In 2008, the country appeared for the first time in a list of 'Top countries in all fields' (ranked by citations per paper). We are now placed 19th, up from 36th place in 2003. I believe that international peer review played a significant part in this development.

Stop playing politics with the peer-review process

mclarke — 2009-08-26T08:58:10+00:00

This text is from a recent Editorial in Nature (460, 667; 2009):

In a depressingly familiar display of irresponsible politicking, the US House of Representatives has taken aim at three studies funded by the National Institutes of Health (NIH). Representative Darrell Issa (Republican, California) introduced an amendment killing the projects on 24 July, during a debate on the NIH's 2010 budget. The House passed the amendment by a voice vote.
Issa was unhappy that the studies looked at substance abuse and HIV risk behaviour, and that the subjects were outside the United States. One focused on Russian alcoholics, another on female sex workers in China and a third on female and transgender prostitutes in Thailand. All three passed muster with NIH peer reviewers, and together would cost about $5 million over five years. Issa wanted that money to be spent at home, and complained that HIV had been heavily studied already. But his reasoning is specious: alcoholism, prostitution and HIV do not respect borders, and any behavioural information that could help slow the transmission of HIV is crucial. Some 33 million people are infected worldwide, and a vaccine is nowhere in sight.
Issa's tactic is not new. Since 2003, conservative House Republicans have tried at least five times to strip funding from peer-reviewed projects that drew their ire. Such meddling threatens to undermine the peer-review process as well as potentially eroding the public's trust that science is above politics.
Also worrying is the House Democrats' acquiescence to Issa's amendment. Democrats facing tough re-election bids hoped to dodge Republican attacks in media adverts in their home districts that might have resulted from opposing Issa. Their assumption is that the amendment can be quietly removed when House and Senate negotiators meet to square their versions of the NIH bill before a final vote on it. But Congress should renounce all tactics that undermine peer review — and cease indulging those who use them.

The wait continues for NIH Challenge Grant applicants

mclarke — 2009-08-18T11:24:51+00:00

From Nature News (Nature 460, 676; 2009), by Meredith Wadman:
Applicants for the coveted Challenge Grants issued by the US National Institutes of Health (NIH) as part of the American Recovery and Reinvestment Act learned the peer-review scores for their proposals late last month. Yet they received little in the way of certainty over whether those scores will translate into money come September, when the NIH will announce which grants it plans to fund.
Competition for the US$1-million, two-year awards is fierce — the agency in Bethesda, Maryland, received more than 21,000 applications, and the NIH director's office will fund only about 1% of these. With ordinary grants, applicants can usually tell if their grant is fundable as soon they receive their percentile score because they already know the designated 'payline', or percentage of fundable applications. The NIH has designated an initial $200 million of $10.4 billion in economic stimulus funds for the grants, but with so many variables at play in allocating the stimulus money, predicting whether a given score will land funding is almost impossible — meaning that those with percentile scores in the mid-single digits are left hanging.
Meanwhile, the burden on the thousands of grant reviewers has, according to some, turned out to be bearable. Gary Johnson, chairman of the pharmacology department at the University of North Carolina, Chapel Hill, told the NIH that he could review up to five Challenge Grant applications. "And they only gave me a couple," he says. "I don't know anyone who was overwhelmed by reviewing, because there was an overwhelming agreement of investigators to participate in the process."

Hoax paper accepted for publication

mclarke — 2009-06-18T14:09:17+00:00

A story published in Nature News online on 15 June, describes how the editor-in-chief of a journal is to resign after claiming that the publisher, Bentham Science Publishing, accepted a hoax article for publication without his knowledge.
From the Nature News story:
The fake, computer-generated manuscript was submitted to The Open Information Science Journal by Philip Davis, a graduate student in communication sciences at Cornell University in Ithaca, New York, and Kent Anderson, executive director of international business and product development at The New England Journal of Medicine. They produced the paper using software that generates grammatically correct but nonsensical text, and submitted the manuscript under pseudonyms in late January. After receiving several unsolicited invitations by e-mail to submit papers to open-access journals published by Bentham under the author-pays-for-publication model, Davis wanted to test if the publisher would "accept a completely nonsensical manuscript if the authors were willing to pay". The manuscript was accepted with a request from the publisher for Davis to pay US$800 to its subscriptions department, based in the United Arab Emirates, before the article was published. Davis then retracted the article.
Bambang Parmanto, an information scientist at the University of Pittsburgh, Pennsylvania, and editor-in-chief of The Open Information Science Journal, told Nature that he had not seen the manuscript or any peer review comments before it was accepted. Nor was he informed that the manuscript had been accepted for publication. "I think this is a breach of policy," he says, adding: "I will definitely resign. Normally I see everything that comes through. I don't know why I did not see this. I at least need to see the reviewer's comments." Parmanto says that Bentham Science Publishing told him that the manuscript had been reviewed by one member of the journal's editorial board. "The peer review didn't work," says Parmanto, who now fears that the journal's publishing system could be open to abuse. "The publisher could take advantage of the fees, and that is why I want to leave," he says.
Mahmood Alam, director of publications at Bentham Science Publishing, told Nature in an e-mail statement that "submission of fake manuscripts is a totally unethical activity and must be condemned." He defended Bentham's peer review process, saying, "a rigorous peer review process takes place for all articles that are submitted to us for publication. Our standard policy is that at least two positive comments are required from the referees before an article is accepted for publication." In this particular case, "the paper was reviewed by more than one person".
Peter Suber, a philosopher at Earlham College in Richmond, Indiana, and a proponent of open-access publishing, is worried that the case could turn people against the author-pays open-access model. "There are many legitimate and rigorous open-access journals that use this same business model," he says.
Further details are provided in two articles at The Scholarly Kitchen blog: an account of the experiment by Philip Davis, Nonsense for dollars; and an editorial, The tip of an Iceberg, by Kent Anderson.
Janet Young, in a comment to the Nature News story, writes: "I've had six requests to review papers from a Bentham Science journal over the last year. The first was for a paper in my field, but I refused as I was very busy at the time. The other five have been for papers in fields I know nothing about - I would have been an utterly inappropriate reviewer had I accepted the requests (I didn't). Each request had the full paper attached to the email, rather than just an abstract. That seems like a very unusual review process to me."
See Nature's news website for the full version of the article, and to add your own comments. You are also welcome to comment here.

Comment 1 (Hoax paper accepted for publication )

Hank Roberts — 2009-10-04T17:06:42+00:00

Can someone post followup pointers on this? It ought to have made much more of a stir, I'd think.

No gender bias identified in peer-review of grant applications

mclarke — 2009-06-01T08:21:58+00:00

NatureJobs reports on the contentious topic of possible gender bias in peer review (Nature 459, 602; 2009). Peer review assesses what is of value in science, yet it has been criticized for biases. One such perceived bias is gender, although evidence for such a bias has been contradictory. A 2007 meta-analysis (L. Bornmann et al. J. Informet. 1, 226–238; 2007; reported at the time in NatureJobs Nature 445, 566; 2007) concluded that women are at a disadvantage in peer review of funding applications. As this study incorporated all known research on this issue, it was suggested to be definitive.
Nevertheless, H. W. Marsh et al. (Am. Psychol. 63, 160–168; 2008) presented conflicting results the following year. This study was comprehensive, based on data from the Australian Research Council (10,023 reviews by 6,233 external assessors of 2,331 proposals from all disciplines), and concluded that the gender of the applicant had no effect on the outcomes of peer review, irrespective of the discipline, gender and nationality of the reviewers, and whether reviewers were selected by a funding panel or chosen by the applicants.
Why should these two studies have conflicting results? To investigate, Marsh and Bornmann now report in NatureJobs how both research teams worked together to reanalyse the data and extend the original meta-analysis. They describe how they applied new, stronger statistical approaches to 66 sets of results representing 353,725 proposals from 8 countries. In this extended study, which will be published in Review of Educational Research, they found no effect of the applicant's gender on the peer review of their grant proposals. This lack of effect held across country, year of publication of the studies included in the meta-analysis, and disciplines ranging from physical sciences to the humanities.
At least for grant applications, all of the co-authors from each of the research teams agree that the weight of evidence suggests that the applicant's gender has no effect on the outcome of peer review, and that these findings are robust and broadly generalizable.
Herbert Marsh is a professor of education at the University of Oxford, UK
Lutz Bornmann is a PhD student at the ETH University in Zurich, Switzerland.
This Peer-to-Peer post is an edited version of their NatureJobs article (which is free to access online).
There is some discussion of this NatureJobs article and the timing of its publication at Nautilus blog.
Note: readers interested in the Review of Educational Research paper in advance of its publication can obtain a copy directly from Dr Bornmann or from the NatureJobs editors.

Websites encourage direct public funding for research

mclarke — 2009-05-21T09:19:50+00:00

The 'SciFlies' project, according to a Nature news story (Nature 459, 305; 2009), will profile scientists from a range of disciplines and the new ideas they want to pursue, or ways in which they would like to expand their current research programme. Website visitors will be able to donate any amount to support the projects they find most interesting or worthwhile.
The website itself states: "We look forward to receiving your application for funding of initial proof-of-concept STEM research projects in the range of $5,000 to $12,000. To participate in this unique online grassroots-funded opportunity, please complete the questionnaire about your project, including details of its possible outcome/impact and profiles of the researchers or research team. SciFlies.org will then depict the project for online users to view and decide if they want to make a charitable contribution in support of the project-funding goal. Once each project’s funding goal is reached, researchers will be notified and the process of funding will be completed based on a mutually acceptable agreement of terms between researcher and SciFlies.org....Projects are competing for support from the general public, so researchers are encouraged to describe their work in appealing and accessible terms, such that users can easily understand the concepts and potential outcomes. Avoid “science-speak,” acronyms or abstract language. A proposal that conveys the researcher’s excitement about a project and its potential, as well as providing insight into his/her personal story, can significantly attract donors."
At this stage, there is nothing on the website about the peer-review or other assessment process, but there are already some projects listed. A full launch is promised for mid-July.
According to the Nature News story, "David Fries, a marine engineer at the University of South Florida in St Petersburg, conceived of and heads the SciFlies effort. His main inspiration was long-standing frustration with a research funding structure that, with few exceptions, offers scientists no intermediate steps on the way to requesting full grant funding." Daniel Gaddy, in a comment to the News story, mentions a similar enterprise called FundScience, which also seeks direct public funding of research projects. FundScience provides a description of its activities here, but although a press release is also provided, there is no information about whether a form of independent peer-review will be used to aid potential donors.

Research council amends controversial grant-funding proposal

mclarke — 2009-05-08T14:36:22+00:00

After a campaign by scientists, the UK Engineering and Physical Sciences Research Council (EPSRC) has softened and delayed its controversial policy to bar serially unsuccessful grant applicants from making funding bids for one year (Nature online News, 5 May 2009). The ban was due to be imposed on 229 researchers starting on 1 June, in an effort to reduce pressure on an overloaded system that currently peer-reviews all grant applications. But eight weeks after it published the policy (see Nature online news 19 March 2009 and Nature 458, 391; 2009), the EPSRC now says that the restriction will not come in until 1 April 2010 — giving scientists more time to change their grant-submission behaviour so that they do not fall under criteria defining repeated failure. And instead of being excluded outright, researchers will be allowed one application during the year.
"We have made these adjustments to address concerns raised by the community — for example, the retrospective nature of [the policy's] implementation," the EPSRC said in a statement. "We've made bold changes to protect peer review, but we're not an insensitive organization."
Peter Main, director of education and science at the Institute of Physics in London, says the EPSRC has listened to criticism and has shown flexibility. "It's the policy that it perhaps should have been in the first place," says Joe Sweeney, an organic chemist at the University of Reading, UK, who set up an online petition demanding the policy be repealed, signed by more than 1,900 scientists.
But some researchers say they are disappointed not to have been consulted more directly beforehand — which might have prevented the EPSRC from introducing the ban in the first place. "It's something of a shame that we had to force them into this policy change," says Philip Moriarty, a physicist at the University of Nottingham, UK.
The EPSRC is keeping a policy introduced on 1 April, to refuse uninvited resubmissions of failed proposals, which it says will cut 20% of applications submitted for review. The exclusion policy had been expected to cut a further 10%.
The EPSRC says that letters intended to warn individuals in April were never sent. "We are an organization that listens to the community," says chief executive David Delpy. "If we can make amendments to help researchers whilst ensuring the overall policy is still effective, then that's in everyone's interest."
Not all scientists oppose the proposals, however. In an online comment to this Nature news story, Nial Wheate writes: "I don't have a problem with the EPSRC policy. Seriously, if you have three grants all ranked in the bottom half in one year, then you need to stop and think about what research you are trying to get funding for. And what's wrong with only submitting 2 grants per year to make sure this doesn't happen to you? The policy makes sense to me."
News story first published online 5 May 2009 Nature doi:10.1038/459020b.

A metric for measuring peer-review performance

mclarke — 2009-04-07T12:22:30+00:00

A guest post from Willy Aspinall
Department of Earth Sciences, Bristol University, Bristol BS8 1RJ UK.

The Nature Editorial (‘Experts still needed’, Nature 457, 7-8 (2009); free to access online) and Harnad’s related Correspondence item on research performance metrics in the 2008 Research Assessment Exercise (Nature 457, 785; 2009) prompt me to suggest that an additional, complementary metric is needed which would measure the accomplishments of research scientists who act as peer-reviewers for journals.
Good reviewing is very time-consuming and, in some ways, just as challenging as authoring an original research paper; time spent doing this well is time removed from one’s own research work. Indeed, the thoughts and comments of a good referee can often represent a fundamental contribution to the science as well as the quality of a published paper, and this input should be recognized, and measured (the American Geophysical Union regularly celebrates ‘excellence in reviewing’ with citations by its journal editors). It is probably fair to say also that tangible good performance in refereeing usually begets ever more requests to review even more manuscripts, with further incursions on the diligent and proficient scientist’s time.
Perhaps a metric for this essential scientific activity of peer-reviewing might be constructed by summing the number of papers refereed by the individual scientist per year, each review being multiplied by the Impact Factor of the journal concerned. As refereeing is usually a solo activity, a metric for this skill, and for the related professional commitment, would be less prey to the shortcomings of performance measurement associated with metrics that attempt to gauge multi-author citations, for instance. Combining a 'refereeing metric' with other citation-related metrics to obtain a more comprehensive performance score for an individual scientist should not be an insuperable problem – and this measure can be pooled, as indicated in the Nature Editorial, with expert evaluation.
Willy Aspinall

Comment 1 (A metric for measuring peer-review performance)

bernd kochanowski — 2009-04-07T14:58:41+00:00

"As refereeing is usually a solo activity,[...]"

When I was working as a young doctor at an microbiological institute the full professor usually let the reviews write by us young ones. I'd assume he was/is not the only one.

Comment 2 (A metric for measuring peer-review performance)

Bjoern Brembs — 2009-04-13T16:41:56+00:00

Obviously, the reputation structure in science is hopelessly out of date. I agree that adding the huge amount of reviewing every scientist performs in his/her career to their reputation is long overdue. I commend the author for pushing this idea.
However, multiplying Thomson Reuter's Impact Factor in any such measure might prove counterproductive.
1. The impact factor is a discredited measure: it's negotiable, irreproducible and an inadequate statistical measure.
2. Creating an incentive only to do good reviews for high-IF journals will skew the overall reviewing process.
On a personal note, I must also object to my reviews for journals which decide not to play the IF game somehow counting less in this metric: I *always* try to deliver the best review I can get, irrespective of the journal.
Therefore, I wholeheartedly support measuring peer-review and adding it to our reputations structure. However, skewing the metric by factoring some JournalRank in the metric is not a good idea. It's an even worse idea if a discredited measure such as the IF is used.

Comment 3 (A metric for measuring peer-review performance)

Massimo Pinto — 2009-04-17T09:08:16+00:00

I agree with Bjoern. There should not be any encouragement to review a manuscript for a high impact factor journal over allegedly "minor" journals.

I would hope to see manuscript refeereing acknowledged in a non-metric, non-measurable way, possibly via letters of reference made by journal editors?
Driving refereeing in the seastorm of metrics is a dangerous route. What would happen next, metrics for those who referee Project Grants?

Comment 4 (A metric for measuring peer-review performance)

Willy Aspinall — 2009-04-24T08:55:29+00:00

I'm happy this post created some discussion. Bernd may be right that I am naive: in my own fields of interest, I have never asked a colleague to help on a review, nor have I ever been asked by one. If the practise he describes is as prevalent as he implies, it surprises me! However, I do subscribe to his reservations about anonymous reviewing, and do not withhold my own identity when this option is offered. Like Bjoern and Massimo, I too try to do the best job on any paper, for whatever journal, because it should represent making a (small) and responsible contribution to science; at the moment, however, the contributions of the papers, and the authors, continue to be scored by suspect metrics, but the poor peer-reviewer does not get even this flawed measure of reputation approval. Massimo's other point about perhaps utilising some non-metric acknowledgment, say by journal editor reference, is an interesting one. As I noted earlier, the AGU acknowledges "Excellence in Reviewing" and doubtless this enhances the reputations of those so applauded, but it carries the risk that all that journal's other reviewers are somehow branded less than "excellent".

Comment 5 (A metric for measuring peer-review performance)

G P S Raghava — 2009-05-19T16:47:34+00:00

Their is an interesting article "Is citation a good criteria",
recently published in "Nature India". In this article "Citation"
commonly used criteria used for evaluating performance of scientist/
departments/nation has been discussed in detail. This article also
discussed h-index, g-index and imact factor of journals. For detail
see following link

Article link: http://www.nature.com/nindia/2009/090515/full/nindia.2009.133.html

Nature Neuroscience experience with peer-review consortium

mclarke — 2009-03-31T11:59:58+00:00

In 2008, the journal Nature Neuroscience joined a newly created community consortium aimed at making peer review more efficient by allowing reviews to be transferred between consortium journals. In its current (April) issue, the editors look back at their experience with the Neuroscience Peer Review Consortium over the past year (Nature Neuroscience 12, 363; 2009).
Journals in the Neuroscience Peer Review Consortium (NPRC) offer authors whose papers are no longer under consideration at a journal an opportunity to transfer reviews of their manuscipts when submitting their paper to another consortium journal. After a year, Nature Neuroscience's experience is similar to that of other journals in the consortium, with only a handful of papers being transferred from Nature Neuroscience to another consortium journal.
Similar to the Nature journals' transfer system, the NPRC system is voluntary for authors and referees. Editors at one journal only know that a paper was reviewed elsewhere if the author chooses to inform them. At Nature Neuroscience, the editors ask referees for permission to release their identities whenever authors ask for their papers to be transferred to another consortium journal. If a reviewer declines to participate, the reviews (comments to authors only) are transferred anonymously.
All the transfers from Nature Neuroscience to date have been to the Journal of Neuroscience, and represent less than 1% of manuscripts that are eventually rejected after review. However, for the papers that were eventually published in the Journal of Neuroscience, the authors reported that the paper had been expedited. Even in the cases where new referees were solicited, authors felt that transferring the reviews from Nature Neuroscience had saved them time and effort.
No papers have been transferred to Nature Neuroscience from other consortium journals.
The Nature Neuroscience editors ask why so few authors are using the NPRC option. They conclude: "Authors may simply not be aware of NPRC or may not know what journals participate in it. Transfer rates may pick up as more authors learn of the consortium. At Nature Neuroscience, we have noticed an increase in the number of referees that state in comments to the editors whether they wish for their identities to be released to other consortium journals or not, suggesting a growing awareness of the NPRC.
It could also be that there are not that many papers that lend themselves well to this process. Many of our authors who have had papers rejected may prefer to take their chances with new referees at another journal, rather than making substantial revisions in response to the concerns raised by our referees. Certainly, our authors appear to be more conservative when deciding to transfer their reviews, preferentially choosing to utilize the NPRC transfer option when the reviewers reject the paper on conceptual grounds and not for technical reasons.
Another factor that influences the success of the transfer is whether the referees allow the release of their identities to receiving consortium journals. Previous reviews are clearly less useful to the receiving journal if the editors do not know who the reviewers were."
Nature Neuroscience concludes that it is premature to gauge whether the system truly could save referees, authors and editors substantial time and effort. The editors encourage authors, referees and readers to share their views, either by email or by commenting here.

Going all out for the right to reply

mclarke — 2009-03-22T09:25:35+00:00

From Nature News, 17 March 2009 (doi 10.1038/458264b):
The Max Planck Society (MPS) in Germany has begun legal proceedings against publishers Wiley International in a dispute over an editorial in the February issue of Human Brain Mapping. The society alleges that the editorial grievously misrepresents it and harms the reputation of one of its scientists. It wants the journal to publish a letter from the society addressing these concerns without delay. Peter Fox, an editor-in-chief at Human Brain Mapping, says that the MPS letter went through normal refereeing processes "in a timely manner", but says he does not know when it will be published. MPS vice-president Herbert Jäckle, deputized to act for the society in this matter, claims that the journal has unfairly delayed the society's right to reply. Fox, a neurologist at the University of Texas Health Science Center in San Antonio, accepted the MPS letter on 11 March, two days after the deadline the society set before taking legal action, says he will publish the letter together with a reply that "rebuts Dr Jäckle's various accusations".
The row originally centred on a dispute over who owned data gathered in the laboratory of Nikos Logothetis at the Max Planck Institute for Biological Cybernetics in Tübingen. Last spring, Amir Shmuel and David Leopold used the data in a paper for a special issue of Human Brain Mapping on spontaneous brain activity — but Logothetis maintains that the data were used inappropriately because they had been gathered for a different purpose, and wrote to the journal asking for the paper not to be published. It was, however, published in May 2008. Logothetis published a scientific rebuttal in another journal, NeuroImage, in January 2009.
Fox and his fellow editors published an extended editorial analysing the dispute in the framework of ethics in the February 2009 issue of Human Brain Mapping. The MPS says that there are numerous factual errors in the editorial's account of events. Jäckle is also incensed by the article's claims that he had given permission to publish the disputed paper. "I was only a mediator in a dispute, not an adjudicator as the editorial claims," he says. "It is not up to the Max Planck Society to permit publication of anything — that responsibility lies solely with the editors."
The MPS has posted a response to the February editorial on the website of Logothetis's laboratory. "The goal of our editorial was to use this conflict to discuss the ethical principles that govern responsible conduct of research and peer review," says Fox, "and thereby to develop guidelines that would prevent future occurrences of this nature."