RAE new metrics are just a 'piece of somnambulism'

HEFCE’s Research Assessment Exercise (RAE) has come under fire. OA champion Stevan Harnad said plans for conversion to new metrics are flawed. He accused the RAE of an “anachronistic piece of somnambulism” in requiring a publisher’s CD of its PDF, instead of asking for deposit of the authors’ papers in their own institutional repository.

“The good news is that there is still time to fully remedy these issues, if only policymakers would take a moment to listen, think it through, and do the little that needs to be done to fix it,” said Harnad.

“It is a good idea to have a national research performance evaluation to monitor and reward research productivity and progress.

“It is also a good idea to convert the costly, time-consuming, wasteful and potentially biased panel-based RAE of past years to an efficient, unbiased metric RAE, using objective measures that can be submitted automatically online,” he said.
Harnad said the biggest flaw concerns the metrics that will be used, as these will change to be based mainly on research funding and citation counts. He also said that newly arrived-at metrics should be cross-validated against RAE panel evaluations.

“Despite its warts, the current RAE panel rankings need to be used to bootstrap the new metrics into usability,” he said. “Without that prior validation based on what has been used until now, the metrics are just hanging from a skyhook and no one can say whether or not they measure what the RAE panels have been measuring.”
According to a HEFCE spokesperson, “There is consensus that the 2008 RAE will go ahead as planned. Detailed arrangements have been developed over a long period of time and are well understood by the HE sector. It would be disruptive to make changes at this late stage. The results of the 2008 RAE will inform funding from 2009-10 and provide a strong baseline for the new framework.

“HEFCE is developing proposals for the future of research assessment and funding beyond 2008. There will be a full consultation on this in the autumn,” said the source.

The spokesperson also believes it will be important for any new system of research assessment to “retain the confidence both of government and the HE sector”.