by BRADLEY SMITH  

The next round of Excellence for Research in Australia will include evaluation of Indigenous research. However, without substantial modification to ERA methodology there is a high risk the ARC will be setting up Indigenous research to fail.

The key challenges are:

* design an evaluation approach that militates against the misapplication of “world standards.”

* provide clarity as to how the three subsets in the definition of Indigenous research are evaluated.

ERA ratings are explicitly evaluated against a putative “world standard.” However, “world standards” are a dubious and potentially corrosive basis to rate Indigenous research. While some Indigenous research has international relevance, a key driver is research addressing challenges and opportunities at community scale and typically published in niche or practitioner focused outlets that do not generate significant international citations.

For this work, research quality is better understood in terms of fit-for-purpose to inform policy, implementation, and practice, including dynamic, generative cultural practice.

Analysis of bibliometric data show Indigenous research publications are significantly lower in key ERA metrics of relative citation impact and percentage of outputs in top citation percentiles than total Australian publications. Unmediated use of world benchmarks for Indigenous research in STEM fields will likely result in all submissions rated below ERA Three, “at world standard.”

Constructing alternate world benchmarks for Indigenous research that restricts the global publication set to a very limited range of journals is likely to produce dirty data through omissions, false inclusions and limited sample.

The challenges for Indigenous research in HASS fields evaluated by peer review are perhaps even more complex in determining “world standard” given community focus, international literature skewed to settler countries particularly the USA, and highly contested views within Indigenous studies fields.

The ARC will need to make explicit how evaluations against “world standards” will be modified, or articulate an alternate framing of ratings.

Indigenous studies is defined in the new research classification (ANZSRC 2020) as research that significantly:

(a) relates to Aboriginal and Torres Strait Islander, Māori, Pacific, and other Indigenous peoples, nations, communities, languages, cultures or knowledges and/or

(b) incorporates or utilises Indigenous methodologies/ways of knowing, theories, practice and/or

(c) is undertaken with or by these peoples, nations or communities.

During the consultation for the new classification, a key motivation of Aboriginal, Torres Strait Islander, Maori and Pacific Islanders was visibility of Indigenous knowledges and research capacity.

Parts (b) and (c) broadly capture these elements, although the extent to which research that “utilises Indigenous methodologies” or is “undertaken with”, will require careful judgement as to degree of applicability.

However, research that “relates to” Indigenous peoples will not only dominate total publication volume but will primarily be authored by non-Indigenous researchers using western methodologies (typically with close engagement with Traditional Owners and Indigenous NGOs).

How the evaluation panels deal with the mixture of Indigenous and non-Indigenous researchers and methodologies must be carefully resolved and explicit in guidelines for submissions and evaluation. Failure to do so may constitute a perverse incentive to skew coding of publications if there is low confidence in the evaluation.

Given the limitations of ERA, the Australian Research Council should consult with relevant Indigenous research expertise such as the Lowitja Institute, to establish a “State of Aboriginal and Torres Strait Islander Research Report” as a discrete summary , that utilises and synthesises data and insights from, for example the ARC’s ERA and Engagement and Impact reports and outcomes from ARC/NHMRC grants with an Indigenous focus.

This would allow nuanced evaluation of

* the efficacy of research through influence on policy and practice (informed by EI),

* impact of research on the research enterprise (informed by ERA), and

* delivery against justifications of grants and contracts.

Bradley Smith is senior research policy advisor at James Cook U

 

Without substantial modification to methodology there is a high risk the ARC will be setting up Indigenous research to fail.