August 2007
The Coming Revolution in Scholarly Communications & Cyberinfrastructure
Incentivizing the Open Access Research Web
Publication-Archiving, Data-Archiving and Scientometrics
Tim Brody, University of Southampton, UK
Les Carr, University of Southampton, UK
Yves Gingras, Université du Québec à Montréal (UQAM)
Chawki Hajjem, Université du Québec à Montréal (UQAM)
Stevan Harnad, University of Southampton, UK; Université du Québec à Montréal (UQAM)
Alma Swan, University of Southampton, UK; Key Perspectives

The UK Research Assessment Exercise (RAE)

The UK has a unique Dual Support System 21 for research funding: competitive research grants are just one component; the other is top-sliced funding, awarded to each UK university, department by department, based on how each department is ranked by discipline-based panels of reviewers who assess their research output. In the past, this costly and time-consuming Research Assessment Exercise (RAE) 22 has been based on submitting each researcher’s four best papers every six years to be ‘peer-reviewed’ by the appointed panel, alongside other data such as student counts and grant income (but not citation counts, which departments had been forbidden to submit and panels had been forbidden to consider, for both journals and individuals,).

To simplify the RAE and make it less time-consuming and costly, the UK has decided to phase out the panel-based RAE and replace it instead with ‘metrics.’23 For a conversion to metrics, the only problem is determining which metrics to use. It was a surprising retrospective finding (based on post-RAE analyses in every discipline tested) that the departmental RAE rankings turned out to be highly correlated with the citation counts for the total research output of each department (Figure 3; 2425).

Figure 3
Research Assessment, Research Funding, and Citation Impact

“Correlation between RAE ratings and mean departmental citations +0.91 (1996) +0.86 (2001) (Psychology)”

“RAE and citation counting measure broadly the same thing”

“Citation counting is both more cost-effective and more transparent”

(Eysenck & Smith 2002) http://psyserver.pc.rhbnc.ac.uk/citations.pdf
Figure 3. RAE citation/ranking correlation: In the Research Assessment Exercise (RAE), the UK ranks and rewards the research output of its universities on the basis of costly and time consuming panel evaluations that have turned out to be highly correlated with citation counts. The RAE will be replacing the panel reviews by metrics after one last parallel panel/metric RAE in which many candidate metrics will be tested and validated against the panel rankings field by field.

Why would citation counts correlate highly with the panel’s subjective evaluation of researchers’ four submitted publications? Each panel was trying to assess quality and importance. But that is also what fellow-researchers assess, in deciding what to risk building their own research upon. When researchers take up a piece of research, apply and build upon it, they also cite it. They may sometimes cite work for other reasons, or they may fail to cite work even if they use it; but for the most part, a citation reflects research usage and hence research impact. If we take the panel rankings to have face validity, then the high correlation between citation counts and the panel rankings validates the citation metric as a faster, cheaper, proxy estimator.

Pages: 1 2 3 4 5 6 7

Reference this article
Harnad, S., Brody, T., Carr, L., Gingras, Y., Hajjem, C., Swan, A. "Incentivizing the Open Access Research Web," CTWatch Quarterly, Volume 3, Number 3, August 2007. http://www.ctwatch.org/quarterly/articles/2007/08/incentivizing-the-open-access-research-web/

Any opinions expressed on this site belong to their respective authors and are not necessarily shared by the sponsoring institutions or the National Science Foundation (NSF).

Any trademarks or trade names, registered or otherwise, that appear on this site are the property of their respective owners and, unless noted, do not represent endorsement by the editors, publishers, sponsoring institutions, the National Science Foundation, or any other member of the CTWatch team.

No guarantee is granted by CTWatch that information appearing in articles published by the Quarterly or appearing in the Blog is complete or accurate. Information on this site is not intended for commercial purposes.