The ARC commissioned expert analysis of its two major research metrics last year. Frank Larkins is not impressed with the outcome

Given the expertise of the review panel “an opportunity has been missed,” Professor Larkins says. Its members “did not do more to address options for solutions rather than just identifying problems,” he writes in a new paper for Uni Melbourne’s L. H. Martin Institute.

Professor Larkins is a concise critic of the Australian Research Council’s flagship programme, Excellence in Research for Australia and the newer Engagement and Impact.

He points to continuing problems in ERA, including a lack of transparency in setting international benchmarks in some STEM disciplines and “clear inconsistencies between metrics-based and peer review assessment methodologies.”

And he proposes two ERAs to measure research quality separately for STEM and HASS disciplines.  Plus, he calls for benchmarking research performance against only the US, UK, Canada and New Zealand.

And he questions whether the new expert group charged with revising the research engagement and impact measure can succeed. “Comparative acceptable assessment performances between universities and the development of a valid rating scale for EI is elusive and unlikely to be achieved.”

The ARC “has much work to do within a very short timeframe,” he warns.

“The methodological barriers to real success, to restore confidence for the stakeholders in the exercises, may be insurmountable.”

Last year he proposed combining ERA and EI and running the resulting metric every five years instead of three (CMM October 15 2020). The next ERA is due in ’23 and EI in 2024 and he now calls for 12-month delays.