Numbers game hots up

The beguiling simplicity of the “impact factor” has made it a figure of supreme importance in research. Journal impact factors, or IFs, measure how often science and social science journals are cited by academics. The measurement of the number of times a journal is cited by researchers in the field has become shorthand for the value of that journal; and funding bodies and employers use citation metrics to assess the productivity of institutions, departments and individuals.

Thomson Scientific dominates the citation metrics landscape with its Web of Science-based citation index. Recently, however, it has faced increasing competition from the likes of Scopus and Google Scholar. The existence of realistic alternatives to Thomson Scientific’s index – and which give different results to it – has thrown the debate on citation metrics wide open.

Thomson Scientific calculates the impact factor of a journal by dividing the number of current year citations into the number of source items published in that journal during the previous two years.

Citation metrics are also used to produce the H Index (a measure of an individual’s publishing activity) and the G Index (a weighted version of the H Index). Crucially, citation metrics will also be used to inform the funding of higher education research under the Research Excellence Framework (REF).

The Web of Science (WoS) is well established with huge coverage. But critics say that WoS is expensive – Google Scholar, by comparison, is free – and its coverage incomplete. They also say that because citation metrics take years to create, WoS cannot identify what is hot right now.

Marie McVeigh, senior manager for editorial development at Thomson Scientific, says: “Citation metrics are inherently looking at something in the past. There is an inherent delay in citation as it takes time to emerge in the literature.”

Lagging, not leading
But Kuan-Teh Jeang, editor in chief of open access (OA) journal Retrovirology, says that metrics designed to measure “previous” modes of publication are an assessment of publishing impact on a largely “Western and developed audience” and are lagging rather than leading indicators.

“Things that seem to be invisible now might prove to be highly impactful,” says Jeang.

“Time has become compressed. What is happening in a year may have taken five years a decade ago. It has become urgent to figure out what are the leading indicators that predict what is going to happen.”

Matthew Cockerill, publisher at OA publishing house Biomed Central, believes the timeliness of the newer indices is an asset. “Google Scholar is wide-embracing and up to date. Scopus adds every new biomedical journal on an annual basis. And Google Scholar adds on an automated basis, which in principle should be quick.”

But he adds: “There are tools available but you can’t do the range of analysis that you can using Scopus and Web of Science. However, this used to be highly proprietary information and now it is really open.”

Thomson Scientific believes that maintaining indexing quality and consistency of citation data is key. “Our focus is on making sure our metrics reflect the scholarly process well,” says Jim Pringle, vice president of product development at Thomson Scientific. He points out that the company supplements its journal citation reports by publishing a hotlist of papers that are emerging as highly cited.

Pringle says Thomson Scientific is watching with interest experiments with other citation metrics from journal ranking body Eigenfactor to download metrics. “With a download or a page view versus a citation in peer-reviewed literature, you are dealing with a different point on the value scale,” he says.

Elsevier’s Scopus citation database has been criticised for going back only as far as 1996.
Niels Weertman, Scopus product manager, explains how this came about: “We formed a content selection and advisory board of 30 individuals from library and research fields in different disciplines to establish the content policy, the selection criteria. They made the fundamental decision to go back as far as 1996 as this met most needs.

“At the same time we recognised that certain researchers – for example in physics literature research – needed to go back further but abstracts and titles are sufficient.”

There is no specified period that must elapse before a journal, which may be open access, is considered for inclusion in Scopus, as with Web of Science, but they will be evaluated differently.

“Publications that have been suggested that have been around a longer time will have a different type of evaluation from, say, a society publisher starting with a new title,” says Weertman.

Scopus introduced an H Index service this year, in response to the fact that 20% of researchers’ searches of its system were to do with people evaluating themselves, their peers and competitors.

When developing the H Index, Jorge Hirsch, who teaches at the University of California, decided that as citation counts were used for research evaluation in faculty recruiting and promotion, as well as in grant allocations, articles that received large numbers of citations should be considered as significant in such evaluations, even when they were not published in high-impact journals.

Hirsch developed the H Index as a metric that could illustrate research achievement.

Weertman says: “Finding information on a particular individual is time-consuming, sometimes needing the disambiguation of the author’s name. Now they can do it quicker.”
Individual metrics have to be used with care, especially if comparisons are to be made.

Weertman warns: “Metrics are one of many things that can help you get a broader or more complete picture. In itself, a metric is not the answer, just a clue. We always say, don’t only look at one metric. Two authors can have the same H Index but one author can achieve it with significantly less work than the other.”

Thomson Scientific also puts health warnings on its metrics. The company says it does not rely on the impact factor alone in assessing the usefulness of a journal, and neither should anyone else. “It is important that people use the metrics well and use them for the right purpose,” says Pringle.

Cockerill at Biomed Central says: “Everybody would like to be able to sum up the importance of a piece of research with a simple number and compare that number with the numbers of other journals. But expecting a single number based on a particular subjective metric to tell you everything you need to know is an unrealistic goal.

“The reality is that people tend to give emphasis to a number and this can be circular so that the decision to submit a piece of research is based on the IF of the journal.

“Evaluation authorities say they don’t attach importance to impact factors but the perception is that IFs are all-important. But people forget about the partial nature of IFs.”

Selection-neutral
Publishers are quick to deny any suggestion that IFs have any effect on their selection of papers. Laurie Dempsey, senior editor of Nature Immunology, says: “We do not pick manuscripts because we think they might have higher cites than other manuscripts under consideration at the journal. Manuscripts that are selected to go out to peer review still have to pass the test of technical quality and significance by the expert referees.”

One of the issues with citation metrics is that they do not thoroughly reflect the range of scientific advancement. Research with a more practical application might be cited less in other research. There is scientific value in clinical trials and individual datasets, yet no-one cites results from them.

There have been moves to include sources beyond journal papers, but there is still a way to go. Scopus publishes conference proceedings publications and 33 million abstracts and Thomson Scientific also publishes conference proceedings as “a way to uncover research ideas as they are presented for the first time - often before publication in the journal literature”.

Beyond the journal
Weertman says: “We want to include other sources, and researchers need to have access to that content. In some disciplines such as science and maths it has been shown that 50-60% of research results are in conference proceedings; while in arts and humanities most research is in a book, not in papers or conference proceedings.”

Citation measures have come under fire but whatever their flaws, they are undeniably relevant.

Research has shown there is a positive relationship between average citations per paper and peer review measures of research performance.

Critics say citation counts are not valid because some citations might be negative rather than positive, but the evidence appears to show that no-one cites uninfluential findings enough to affect aggregate measures.

Other criticisms range from the distorting power of citation clubs (where researchers band together to cite each other and raise their citation rate) to overlooking sleeping beauties (papers that do not attract attention when first published but gain relevance and citations many years later, which is too late to feature in Thomson’s IF). The sleeping beauty scenario is thought to be rare and possibly balanced out by those rare papers that peak early and are never cited again.

The increasing complexity of the metrics landscape should have at least one beneficial effect: making people think twice before bandying about misleading indicators. More importantly, it will hasten the development of better, more open metrics based on more criteria, with the ultimate effect of improving the rate of scientific advancement.

FUNDING FACTOR: METRICS AND UK SEARCH BUDGETS
In the UK much of the citation metrics debate has focused on the proposed use of quantitive metrics to underpin future research funding under the Research Excellence Framework which will follow the current Research Assessment Exercise (RAE) 2008 funding programme. Consultation on the assessment and funding of higher education research is taking place now, with a deadline of 14 February.

The Higher Education Funding Council of England (HEFCE), which uses the RAE to determine future research funding, acknowledges there are limitations to a metrics-based approach but says it intends to counter the drawbacks by using a wide range of metrics.

It is also careful to say that its application of the principle of quantitive metrics does not entail the grading or ranking of journal titles (journal impact factors) nor of individual researchers.
Jonathan Adams, director at Evidence Ltd, which has contributed metrics research to HEFCE, says bibliometric evaluation may even boost research publication, citing the experience of the Netherlands when it switched to bibliometric evaluation. In an article in Research Fortnight, he points out: “Dutch researchers responded by improving their publication practices and output, and citation rates shot up, placing the Netherlands’ index values among world leaders’.”

Measure the metrics
Cognitive scientist and open access (OA) evangelist Stevan Harnad also welcomes the introduction of metrics to supplement and eventually substitute for panel review, but he believes HEFCE must test and validate many potential metrics against the panel reviews in 2008.

He says: “The candidate metrics must go beyond just ISI journal impact factors, or even article/author citation counts. Non-ISI citation data (such as Google Scholar), download data, co-citations, and many other candidate metrics should be tested and validated against the RAE 2008 panel reviews, discipline by discipline.

“Open access looms large in both the generation and evaluation of metrics. RAE/HEFCE still has not made the link. Once OA self-archiving is mandated UK-wide and worldwide there will be an unprecedentedly rich and diverse set of OA metrics to test and validate.”

Rama Thirunamachandran, HEFCE director of research and knowledge transfer, says: “The reason we are preparing to make this quantitive-based is that the research sector is much more mature and the RAE was becoming much more complex and a significant burden on the universities and funding council. It is a way of simplifying this as at least in the science subjects you can do this by quantitive measures.”

He believes peer review versus metrics is a false polarity: “Bibliometric measures are based on measures that have been peer-reviewed. Not just journal articles but student results are validated by external examiners; even business investment is a form of peer review as they would not give money for something that wasn’t good.”

Thirunamachandran acknowledges that, at least at the early stages of REF, there will be a reliance on Web of Science data, but over time it will consider bringing in other sources and institutional repositories. “At the beginning we need to look at robust information which is based on peer review. With OA some are peer-reviewed, some are not. The Web of Science database has a good coverage. Scopus is potentially a major competitor for the Web of Science and will drive up the quantity and quality.”

Thirunamachandran says that the HEFCE is aiming to come up with a measure that is at least as robust and acceptable as the RAEs that have gone before.

Model misbehaviour
Adams says: “There will need to be a lot more modelling to explore the relationship between bibliometrics and other measures of quality; the argument is not yet convincing enough to inspire confidence.

“There is a serious risk that an overly complex model, with many weighting factors and other balancing tricks, will become opaque and the real influence of the quality metrics will be hard to follow. That will then undermine researcher confidence in the process.

“Another problem is that people will start to respond not to the objective – high-quality research – but to the signal, a perceived set of indicators in their area. The signal then starts to drift off target and after that we lose our ability to know whether the UK’s research is still any good.”

Like this story? Spread the news by clicking below:

 del.icio.us     Digg this     reddit!