Journal Ranking — the Second Incarnation

Australian academia is again in the throes of a journal ranking exercise.  We went through this last year in preparation for the previous government’s Research Quality Framework (RQF), but the new government wants to redo things for its Excellence in Research for Australia (ERA). 

Again the journals must be placed into four tiers — the top 5% into tier A*, the next 15% into tier A, the next 30% into tier B, and the last 50% into tier C.  However, this time the Australian Research Council has done the ranking; they have re-ranked all the mathematical sciences journals.  This has involved

  1. a substantial reduction in the total number of journals that the ARC will currently accept (they have, of course, correspondingly reduced the number of journals we can place into bands A* and A); and
  2. the use of impact factors to rank journals in applied mathematics and statistics, and apparently also in mathematical physics.

Regarding the number of “research outlets” that the ARC is currently willing to regard as mathematical science journals, let me try to give you a sense of the scale of the changes.  According to my calculations, the new ARC list of ranked journals allocates 538 journals to pure mathematics, 211 journals to applied mathematics, 28 journals to mathematical physics, and 169 journals to statistics (including probability), making a total of 946 journals for the mathematical sciences.  However, the list produced last year allocated a total of 1369 journals to the mathematical sciences.  (These journals were a subset of those currently covered by MathSciNet.)  That is 45% more than the ARC list.  On this basis, and making some assumptions about uniformity of distribution among the four research areas, we should expect the ARC’s list to contain only two-thirds the number of journals in tiers A*, A and B as the previous list; and it does.

A number of important journals have disappeared entirely from the earlier mathematical sciences list; they appear nowhere on the ARC list.  In my field, the journals Bioinformatics and Biostatistics are extraordinary omissions.  One reason for this change (although not one that applies in the above two cases) may be that some of the ARC’s advisors have an aversion to journals that are not covered by the on-line citation index, the Web of Science (usually accessed through the Web of Knowledge).  Another major reason for the reduction in the number of journals on the ARC list has been the ARC’s resistance to listing, in two or more different categories (or, equivalently, against two or more four-digit Field-of-Research, or FoR, codes), the journals where mathematicians publish.  For example, a journal where mathematicians publish their work in mathematical biology, and which is classified as a biological sciences journal, might be ranked low by biologists but high by mathematicians, but will have to keep the low ranking as far as mathematics is concerned.  In the previous list a journal could be ranked differently by the two groups.

Apart from reducing the number of journals we can have, and therefore also the number of journals we can have in the A* and A categories, the disallowance of dual rankings strongly inhibits multidisciplinary work in the mathematical sciences.  The fact that mathematicians and statisticians publish in many non-mathematics journals should be seen as a strength, not a weakness, especially in an age where first-rate work in science and technology is increasingly multidisciplinary.  Therefore dual ranking should have been encouraged.

The ARC’s use of citation-based journal-ranking methodology is also a serious problem.  The international community of mathematical scientists has expressed significant concern about the use of citation data to infer journal rankings, or to determine the performance of mathematical scientists.  In particular, the International Mathematical Union, the International Council on Industrial and Applied Mathematics, and the Institute of Mathematical Statistics have recently produced a joint report addressing the shortcomings of citation analyses; see also the IMU’s press release.

I’ll spare you further details of the problems we are experiencing with the rankings, and point out here only that the National Committee for the Mathematical Sciences is again using four subcommittees, representing theoretical mathematics, applied mathematics (including numerical and computational mathematics), mathematical physics and statistics & probability, respectively, to revise the ARC’s list.  The draft statistics list has already been posted.  [The other draft rankings can be found at this page.]
About these ads

7 Responses

  1. It seems to me that an outcome of this exercise will be that certain avenues of publishing research will be essentially off-limits to those trying to establish a career in mathematics and those trying to win ARC funding – though I suppose that this has always been the case to some extent. For example, I was recently asked to consider a conference proceedings my supervisor will be editing later this year for the publication of a paper I have in preparation. The problem with this is that, as far as I am aware, conference proceedings will carry little or no weighting as far as the ARC is concerned. I assume that the weighting that the ARC applies will eventually become used in assessing the research potential (read: grant-winning potential) of job applicants for mathematics departments. I am not suggesting that where one publishes has never mattered – of course it always has – but if, as Peter details above, some important journals are completely left off the list, some people might have their research output and potential unfairly measured.

    Another point I have been pondering is whether the Journal of the Australian Mathematical Society will experience a decline in submissions from Australian mathematicians due to its B ranking. I don’t know that this would necessarily be the case, as one would expect that mathematicians will usually submit their work to as prestigious a journal as possible – with or without an ARC ranking system in place – but I do wonder whether there could be some effect in this way.

    I suspect others reading this will know more about these issues than a snotty-nosed grad student like me, so all comments will be appreciated.

    Philip Brooker

  2. This kind of system could easily turn Australian university departments into “mutual admiration societies”. A recursive loop of you get funded by ARC if you are inside the system, you get published in the right journals because you were funded by ARC, you get funded by ARC because your work was published in the ‘right’ journals, ….

    Unfortunately, private industry and students don’t give “a rat’s ass” about ARC ratings, so the universities become increasingly irrelevant. Some would argue that this already pretty much the case, so don’t complain about further falling enrollments.

  3. I had a look at those ARC journal rankings and I encourage mathematicians worldwide to take a good close look at them. The bias toward traditional British empire journals is amazing, likewise the bias against journals from Eastern Europe and Asia. The message it sends to the world is: Regardless of your achievements if your ethnicity is not of the preferred background, your work carries no academic merit in Australia. I am sure this is not the message most Australian academics want to send, but that is certainly how it can be interpreted.

    • I don’t think the list can be interpreted that way at all. There is very good reason for the ‘bias’ towards “traditional British empire journals”, namely that the advent of widely-published periodic journals has coincided with the dominance of Western mathematics in the last few centuries. Other cultures have had golden ages of mathematics in centuries past, but they didn’t seem to produce any journals that endure to this day. So it is only natural that journals started in the West are dominant at the moment.

      Moreover, it is suggested that if someone is not of the preferred ethnicity, the their work carries no academic merit. That is absolutely false. The ARC does not evaluate research proposals on the basis of ethnicity, and the fact that so-called traditional British empire journals (there are plenty of high-ranking Continental journals as well) rank highly does not discriminate on the basis of ethnicity, because such journals do not only accept submissions from people from some preferred ethnic background or nationality, but rather from people anywhere in the world. To see that this is the case you only need to look at the contents web-sites of journals online. Ever since the advent of Acta Mathematica more than one hundred and twenty-five years ago (long before ARC journal rankings existed!), journals have become increasingly international and do not just publish submissions from people born and working in the country in which the journal is published. If that wasn’t the case, then Australian mathematicians would only publish in Australian journals and the ARC ranking list would be less than 1% as long as it is now.

      Even if it is the case that higher-ranking journals do have a higher proportion of articles from researchers based in certain countries, this will be because that country has, on the whole, a stronger research culture than other countries. As noted above, rich Western countries have better funding and stronger research cultures in general. That’s not racist, it’s just a fact of life. This may change in the future, just as dominance (mathematically speaking) shifted from Germany to the United States with people fleeing the Third Reich and then the Cold War.

      I think that if mathematicians world-wide did look closely at the list, they would likely quietly disagree with a handful of the rankings, but agree on the whole that the ranking is about right. Whether or not a journal ranking is an appropriate metric for assessing a research proposal is another matter altogether. The main criticism of this journal ranking list is/was primarily that too many journals and other publications were left off the list. This might be addressed as the process is fine-tuned, I guess.

  4. [...] discussed another aspect of the ERA, namely the journal ranking exercise, in an earlier blog post [...]

  5. [...] university (or Law School) rankings, do they begin to skew journals to meet the ARC criteria, or to influence authors when they are thinking about [...]

Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: