Measuring Research Output

Senator Kim Carr, has announced "Improvements to Excellence in Research in Australia" (ERA). This is the process by which researchers and the organisations they work for (mostly universities) are judged. Much of the assessment of researchers is based on what they publish, in books, journals and at conferences. This process allocates a rating to each book publisher, journal and conference, on the assumption that each publication is of that standard. This process has been controversial, particularly the "journal quality indicator". The minister has indicated that the A*, A, B and C ranks will be removed for 2012.

As someone who very occasionally publishes academic work, the process used has been somewhat perplexing. There was no easy way I could find to see what the ranking of the journals and conferences in my field were, so I target the most appropriate. There is a ERA 2010 Ranked Conference List - Excel Format (411KB) and a ERA 2010 Ranked Journal List - Excel Format (5.27MB) - ZIP File (584KB). But these are large unwieldy spreadsheets of data.

It is not clear what the process to be used in the future is. My suggestion would be to make the process of building research reputation explicit and able to be audited. With this, anyone could rate the work of anyone else. The value given to the rating would be weighted by that person's own net ranking. Thus if I rate highly in a particular field, then my assessment would count more than the average person. My ranking would depend on how others ranked my work.

Such a system would be at risk of abuse, by a group of people colluding to rank each other highly and mark down their competitors. However, this would be no worse than the current system, where such corruption is hidden behind double blind review processes. Instead any unusual patterns could be detected automatically by software and made public.

In addition the idea of a journal being restricted to a particular topic could be eliminated. The idea that papers must be published in a journal specialising in one area is a hangover from the era of paper publishing, which has now past. Instead a paper can have several subject areas associated with it and be found by the reader with a search.