HASS disciplines need numbers they can rely on

Citation metrics demonstrate that education researchers in Australia really rate on a global scale – but that is not what the last Excellence in Research for Australia ranking revealed. Anna Sullivan (Uni SA) suggests peer reviewing is a problem.

“It’s likely that the peer reviewers who evaluated the quality of a selection of research outputs did not benchmark accurately in previous evaluations. But this is not unexpected given the lack of clarity around the benchmarks,” Professor Sullivan suggests for the Australian Association for Research in Education.

This really matters as the Australian Research Council asks for advice on how to assess performance for next year’s ERA, Professor Sullivan suggests.

And it really, really matters because in ERA ’23 the ARC will include more ratings scales, to demonstrate how Australian research fields compete against the global absolute elite. This may not be that much of a problem for sciences ranked by citations but the last ERA suggests it could be bad for peer-reviewed disciplines, like education.

And history.  In ERA 2018 departments that expected to do well (high-profile hires, prestige publications) didn’t. The problem, some suggested, was that peer reviewers are dedicated followers of historiographical fashion (CMM May 29 2019).

So what is to be done. Professor Sullivan asks, “why isn’t the ARC including citation metrics in the suite of benchmarks for all disciplines? It might offer a checking balance to moderate peer reviews.”

Or, as Sean Brawley (Uni Wollongong)  explained in CMM this week,

“The metrics-driven disciplines essentially have standards through which their College of Experts make data-informed judgements. … In the peer review disciplines, however, we rely on the peer reviewers and the College of Experts having the same idea of what quality looks like (across a diverse range of disciplines) and then the assigning of the final ranking is contested and debated.”