by DANNY KINGSLEY and HERO MACDONALD
The Review of the Australian Research Council Act represents a once in a generation opportunity for Australia to modernise and right-size its approach to research assessment. This is our chance to systemically enshrine principles of Open Research at the core of our national strategy, incentivising research practices that prioritise reproducibility, integrity and equity of access to Australia’s research.
Assessment creates culture. What gets measured tells us what matters, it both embodies and shapes our values and drives our behaviour. In defining a new national approach, we have an unprecedented opportunity to incentivise a new set of research and publication practices that will empower good research by design and accelerate translation and impact.
Globally, we are seeing growing calls to support a fundamental change in the way we think about research and research assessment. In the last few weeks alone, the Group of Seven science and technology ministers have restated their support for research assessment approaches that incentivise and reward open science practices [1], and the European Council has called for transparent, equitable and open access to scholarly publications [2]. Australia has an opportunity to not just join these conversations, but take a leading role through developing a national Open Research strategy to inform our new approach to assessment.
Perverse incentives
Much has been written about the perverse incentives inadvertently created by the dominant publisher-led, metrics-based, approach to research assessment, and indeed the many problems with a largely profit-driven scholarly publishing landscape [3].
We are counting what we can, rather than what we should be counting. Metrics are easy to measure, and expanding what is defined as markers of quality will make assessment more challenging, but this will be necessary for us to migrate to a more sophisticated and purposeful assessment regime. In the design of any new data-driven approach, we need to be very careful to select measures that drive good research and align with our values and aspirations for societal impact.
Historical approaches to research assessment, which have disproportionately relied on quantity-based journal-level metrics to make judgements about “quality” at scale, have unintentionally encouraged a range of research practices that do not align with our national aspirations for impact or the enablement of “good science.” The publish or perish culture with its focus on the article as a measure of success and using journal prestige as a proxy for quality has resulted in proliferation of publication, which has disadvantaged interdisciplinary and applied research and has created a poor research culture.
We urgently need to consider how our future assessment system will measure and reward not “just where research is published, but all aspects of accessibility, quality and integrity of research” [4], adequately and accurately acknowledges the contributions of different actors throughout the research process, and prioritises access and translation to increase value to the community.
Open Research enables impact and integrity by design
The best mechanism to incentivise both quality and impact is by enshrining a commitment to Open Research at the core of the assessment framework. Open research accelerates impact by enabling rapid and transparent engagement with research [5].
An Open Research approach also reduces the potential for poor practise to occur without detection – protecting the integrity of the research itself. Removing the focus on where research is published allows for a myriad of benefits. By valuing outputs in addition to publications there is an opportunity to recognise and reward different actors in the research process [6].
Despite calls, [7], Australia does not currently have an Office of Research integrity. Indeed research integrity appears not to be a high priority in our institutions, given that a recent survey found that Research Integrity Advisors in many Australian institutions have received no training or were not aware they were an advisor [8].
Existing frameworks such as the San Francisco Declaration of Research Assessment and the Leiden Manifesto provide an excellent starting point. We can leverage work underway internationally, such as the European Agreement on Reforming Research Assessment [9]. We do not have to start from scratch.
Let’s boldly go
We have to be very careful not to further re-entrench perverse incentives in any future assessment process. In rethinking our approach to national research assessment, we have the potential to not just follow international practice, but lead the world in the design of a new approach to assessment enshrining the principles of quality, integrity, openness and sustainability at its core.
Let’s not waste this golden opportunity to develop a national Open Research strategy and an assessment system that truly rewards integrity and impact.
Dr Danny Kingsley is a Scholarly Communication Consultant, and Visiting Fellow at the Australian National Centre for the Public Awareness of Science
Hero Macdonald is University Librarian at Deakin University
[1] G7, 12-14 May 2023 “G7 Science and Technology Ministers’ Communique” https://www8.cao.go.jp/cstp/kokusaiteki/g7_2023/230513_g7_communique.pdf
[2] European Council, 23 May 2023, “Council calls for transparent, equitable, and open access to scholarly publications”
[3] The Conversation, 12 September 2019, “‘There is a problem’: Australia’s top scientist Alan Finkel pushes to eradicate bad science” https://theconversation.com/there-is-a-problem-australias-top-scientist-alan-finkel-pushes-to-eradicate-bad-science-123374
[4] Virginia Barbour, The future of academic publishing: disruption, opportunity and a new ecosystem. doi: 10.5694/mja2.50265 https://www.mja.com.au/system/files/issues/211_04/mja250265.pdf
[5] BMC Research Notes, 20 September 2022 “Promoting trust in research and researchers: How open science and research integrity are intertwined” https://bmcresnotes.biomedcentral.com/articles/10.1186/s13104-022-06169-y
[6] Katherine Christian, Jo-ann Larkins2, Michael R. Doran, 7 December 2022, The Australian academic STEMM workplace post-COVID: a picture of disarray https://www.biorxiv.org/content/10.1101/2022.12.06.519378v1.full.pdf
[7] The Conversation, 3 Feb 2022 “Australia needs an Office for Research Integrity to catch up with the rest of the world” https://theconversation.com/australia-needs-an-office-for-research-integrity-to-catch-up-with-the-rest-of-the-world-176019
[8] CMM, 11 May 2023, “Who knows about research integrity” https://campusmorningmail.com.au/news/who-knows-about-research-integrity/
[9] Science Europe, 20 July 2022, “European Agreement on Reforming Research Assessment” https://www.scienceeurope.org/our-resources/agreement-reforming-research-assessment