Sunday, November 07, 2010

Evaluating Research by Publication

Last month I provided some notes from a forum on measuring research output at the Australian National University in Canberra (October 18, 2010). The transcripts of the presentations are now available:

Overview: Research Excellence Evaluation, Colin Steele, Emeritus Fellow, ANU.
The Australian Research Council -ERA National Perspectives, Andrew Calder, Director, Research Performance and Analysis, ARC.
The ANU Experience, John Wellard, Director ANU Research Office.
Perverse and Beneficial Outcomes From Research Assessment, Professor Andrew Cockburn, Director, College of Medicine, Biology and Environment, ANU.
Scholarly Publishing Economics: an International Research Perspective, Professor John Houghton, Victoria University.
Future Publishing Metrics – The Way Forward?, Dr Danny Kingsley, Manager Scholarly Communications, ANU.
Research Impact – the Wider Dimension, Dr Claire Donovan, Lecturer in Sociology, ANU.

Also Colin Steele has pointed out that the event "RESEARCH ASSESSMENT AND PUBLICATION METRICS - THE BEGINNING OR END OF AN ERA?", was organised by the ANU Emeritus Faculty, not the university overall, as I suggested.

Most of the "transcripts" are simply text, but Andrew Calder's "The Australian Research Council -ERA National Perspectives" is a set of slides. Here is a text version of the slides. Note that the FoR codes referred to are the Australian and New Zealand Standard Research Classification (ANZSRC) and "Scopus" is a list of journal articles and citations:

Excellence in Research for Australia

Andrew Calder

Director – Research Performance and Analysis

Australian Research Council

Australian Government

General ERA Principles

Unit of Evaluation is the four‐digit ANZSRC Field of Research code (ie. 157 possible Units of Evaluation); evaluation occurs at the two‐digit level as well
Evaluation by Research Evaluation Committees in discipline clusters; eight clusters in total
There is a minimum level of output for a discipline to be considered ‘research active’ for evaluation in ERA
Evaluations informed by a ‘dashboard ‘ of discipline‐specific indicators
Some peer review of outputs accessed through institutional repositories in some clusters

ERA Process Overview

Metrics Profile 1	Metrics Profile 2
Metrics Profile 3	Metrics Profile 4
Metrics Profile 5	Metrics Profile 6
Peer Review (if included)

↓

Research Evaluation Committee

↓

Final report

Note - There are no weightings!

Mythbusting ‐ Ranked Outlets

Only one of a number of unweighted
indicators on the “Dashboard”
Ranked conferences are essential for ICT,
Engineering & Built Environment
Note discipline‐specific practices

Role of the ranked outlets list in ERA

Ranked journals
Ranked conferences
- FoR assignments
- Ranks used for ranked journal/conference profile
- FoR assignments used to derive discipline specific benchmarks for citation analysis

Developing the ranked journal list

Initial development by Learned Academies and Peak bodies

↓

Public consultation (June‐Aug 2008)

↓

Expert Review of the public feedback

↓

Omitted journal public feedback (Aug‐Nov 2009)

↓

Final expert review of consolidated list (Dec 2009 – Jan 2010)

Parameters for inclusion in the lists

academic/scholarly
publishes original research
peer reviewed or equivalent process
active during the ERA reference period (2003‐2008)
has an ISSN
able to withstand international scrutiny

The 2010 Ranked Outlet Lists

The 2010 lists are fixed for this ERA evaluation
ERA 2010 Ranked Journal List
- 20,712 journals included
- Over 88% agreement on ranks and FoR
ERA 2010 Ranked Conference List
- 1,952 conferences included

ERA 2010

Background Statement

Volume and Activity	Staffing Profile Research Outputs	Journal articles and listed conferences are apportioned against the FoR codes of the ERA Journal and Conference Lists. All other outputs and eligible researchers can be assigned and apportioned to up to 3 four‐digit FoR codes of the institution’s choice. Staff must be employed on the census date to count towards ‘FTE’.
Ranked Outlets	Ranked Journals Ranked Conferences	All disciplines use Ranked Journals as an indicator; only some FoRs use Ranked Conferences. Where Ranked Conferences is applicable, institutions must apportion against the FoR codes of the ERA Conference list. Where an output of a listed conference does not belong to the listed FoR, institutions can submit the output as a non‐listed conference and assign up to three FoRs of their choice.
Citation Analysis	Relative Citation Impact (RCI) against world and Australian benchmarks RCI Classes Centile Profile	Only applies to journal articles Low volume threshold is 50 apportioned indexed journal articles. The citation supplier for 2010 is Scopus. The citation census date is 1 March 2010.
Peer Review	Peer review	Applies to a range of outputs including journal articles, books, book chapters, creative outputs, etc. Low volume threshold is 30 apportioned outputs (any type). Institutions nominate 20% of total output for a FoR for peer review.
Esteem Measures	Editor of prestigious works of reference Recipient of, Category 1 Fellowship or Australia Council Grant or Fellowship Membership of, statutory committee and Learned Academy	Institutions can select up to 3 four‐digit FoRs (apportioned) for each individual esteem measure. Esteem must be linked to a listed staff of the institution. Individuals researcher cannot be identified through the esteem measures.
Research Income	Category 1‐4	Institutions can select as many four‐digit FoRs (apportioned) for each income item submitted. Number of grants is collected for Category 1 income only. FTE is used as a denominator for all Categories. Category 3 income is disaggregated into the 3 subcategories (Australian, International A and B).
Applied measures	Patents Commercialisation Income Plant Breeder’s Rights NHMRC Endorsed Guidelines Registered Designs	Institutions can select up to 3 four‐digit FoRs (apportioned) for each applied measure submitted, except for commercialisation income where there are no limit on the number of four‐digit FoR codes submitted. Applied measures are linked to the institution, not the individual.

Net Traveller