The ARC is reviewing its two research metrics

The Australian Business Deans Council puts issues with both Engagement and Impact and Excellence for Research in Australia on the agenda in its submission, including;

ERA

*  citation analysis “is no substitute for peer review on the quality of scholarship”

* “world standard” in ranking performance “is not robustly defined”

* “the rating scale rewards publication in international journals, some of which do not appear to be as receptive to submissions based on Australian data as they could be”

EI

* “has had a broadly positive impact” in accounting for non STEM research, “not adequately captured” by ERA

* impact criteria should be clearer

* using narratives is “not sufficient for assessing engagement”

Overall, the ABDC wonders about the effort submissions require and the way ERA increases competition, rather than collaboration, between disciplinary colleagues in different institutions

The Regional Universities Network suggests ERA meets objectives with government and universities, “but is not broadly recognised by industry or international students.” As to EI, “the numeric part of the assessment isn’t useful with respect to end-users, but the narrative is.”

* overall RUN is ambivalent about ERA, suggesting it has made research “more competitive” and “helped universities focus on strengths”. However, it consumes “considerable” resources

* and it does not like peer review for ERA, “it isn’t yielding reliable or repeatable results.”

But RUN is rare among submissions reported to date, suggesting, “word standard is appropriately recognised under the current rating scale” (and) “cannot be moved.”

As to EI;

* “the numeric part of the assessment isn’t useful with respect to end-users, but the narrative is. The discourse on the narrative with researchers has been helpful in encouraging them to engage with end-users. “

But while RUN approves of much of the methodology and its outcomes, it is scathing about its impact on Australian universities as a whole, “EI has had little impact. It isn’t linked to funding, and is poorly designed. It isn’t designed to demonstrate what it is meant to.”

Get the word out

The ARC plans to release submissions to the research metrics review after it is out, which seems a bit late for a debate. So, CMM will report and/or link to, as many submissions as it can – send them in people.