There is an interesting new report “Citation statistics“, jointly produced by the International Mathematical Union (IMU), International Council of Industrial and Applied Mathematics (ICIAM), and the Institute of Mathematical Statistics (IMS), on the use and abuse of various citation statistics (such as impact factors and h-indices) as proxies for research quality. (One of the authors, incidentally, is Peter Taylor from the University of Melbourne, not to be confused with Peter Taylor from the Australian Mathematics Trust.) The press release for the report is available here. The basic message is that these statistics can supplement expert judgement of the quality of one’s research, but cannot substitute for that judgement, despite being more a “objective” metric, as they are subject to various artificial distortions. (For instance, a typical paper in the life sciences is cited six times more frequently than one in maths or computer science, due to a variety of factors, including the different academic cultures of these disciplines.)
Of course, expert evaluation by someone knowledgeable in the subject matter is a scarce resource, and it is still very tempting to rely on these statistics in the absence of such judgment. I once was involved in applying for a large Australian grant that was open to all sciences. One of the reviewers commented that the proposers (who were all mathematicians) had significantly fewer publications than those from competing proposals, particularly those in the life sciences (though my own publication count of 150 or so papers was deemed “acceptable”). While statistics such as impact factors are intended to remove some of the distortions coming from using raw publication count as a measure of research quality and output, they are still far from perfect, especially when it comes to comparisons across disciplines. (For the record, our proposal was not funded, though this was probably a result of many other factors than the above comment.)
[Via The Funneled Web and the Australian Mathematical Society.]
Filed under: Research, Studies | 7 Comments »