by JAMES GUTHRIE and JOHN DUMAY

A recent study of the impact of neoliberalism on Australia’s public higher education system explores how calculative practices and performance metrics have become the new language of universities.[1] It suggests Australian public universities are in a sorry state, brought about by a focus on the means rather than the end. By privileging revenue generation and expenditure over education’s potential to contribute to the nation and society, it seems that universities have lost their primary purpose. We call for a radical rethink of the public sector university mission for the ultimate benefit of the Australian community.

Recent articles in Campus Morning Mail explore the national higher education data collection system[2] and the global ranking systems that pressure academics to achieve so-called KPI performance indicators for research, teaching, commercialisation and outreach.[3]

As academics and accountants, we have first-hand experience of these calculative practices and how rankings are produced. Here we tell our recent story ‒ the story of James and John ‒ to illustrate.

Below is a recent news story on the Macquarie University website.

***

” Research Successes and Accolades

Macquarie University has been recognised as the best Business and Management University in NSW. Research.com studied the work of almost 6,000 business and management researchers from more than 500 institutions, concluding Macquarie University and its staff ranked first in NSW, third nationally and 43^rd worldwide.

Macquarie Business School Professor James Guthrie and Professor John Dumay were among the “Top Business and Management Scientists” recognised in the rankings. Congratulations to all at MQBS on this wonderful achievement. View a snapshot of how we ranked in Australia.”

***

We were pleased to be recognised but also curious. How are these rankings calculated? Research.com is a for-profit company selling its research consulting services to universities, governments and head-hunters. Not only are we uncomfortable with a private organisation profiting from our individual research outputs, but the data also it relies on, at least concerning our research, is incorrect.

Research.com claims, “Our top scientists ranking is a reliable list of leading scientists from the area of Business and Management, based on a meticulous examination of more than 5972 scientists on Google Scholar and Microsoft Academic Graph”. However, when we looked more closely at the data, we found some disturbing anomalies.

First, Microsoft Academic Graph was discontinued as of 31 December 2021 and therefore is no longer a data source we can interrogate to verify research.com’s “meticulous examination”. Second, there is a significant discrepancy between Google Scholar data available in December 2021 and what is reported.[4] It appears that half of John’s citations are missing, especially his highest cited paper[5] James’ data is also wrong as it includes the publications of another academic called James Guthrie.[6]

Regardless, Macquarie University was keen to spruik its success, particularly that of the Department of Accounting and Corporate Governance, listed as the top Accounting Discipline in Australia (according to League of Scholars, on the basis mainly of James’s and John’s citations).

In 2020, John was ranked as the top accounting researcher in Australia by the League of Scholars. However, in 2021 the data used to assess the top accounting researcher was incorrect, and he fell in the rankings, despite having substantially more citations than in 2020. According to the League of Scholars, the methodology for the top researcher is, “the leading Australian-based researcher in a field is the individual whose papers published in the 20 top journals in their field (measured by H-index) over the past five years have had the most citations by other researchers. This is a measure that filters for both quality and impact”.[7] Nevertheless, comparing John’s Google Scholar profile against that of the 2021 nominee, John has achieved more citations in a year than that of the 2021 nominee’s lifetime total. Similarly, comparing James’s results using these criteria exceed those of the 2021 nominee.

The League of Scholars data makes verification of data difficult because of the following disclaimer on their website

“Please note that this public version of the database has not been updated recently and does not cover all variables. We update our data regularly and use the updated data in our analysis work. This version is only for you to get a flavour of the data that we use.”

John P A Ioannidis of Stanford University, and colleagues, provides an example of a rigorous and transparent ranking of scholars, which is peer-reviewed, and all the data and analysis for every person included is openly available in the published raw data.[8] There is no commercial enterprise behind the results.

Regardless, the status of such rankings cannot be understated. Elsevier claims that over the past 18 years, international university rankings have grown in visibility and prominence. “They can influence:

• how your government measures research excellence for your institution
• whether an undergraduate or graduate student (and their parents) opt for your university
• why a company selects you as a partner
• whether a funding body invests in research at your university.” [9]

However, we have seen no empirical evidence to support this in the Australian context, especially government funds allocated based on rankings for research and teaching.

Our point here is not to complain about how these organisations got our rankings wrong. It shows how the calculative practices used to judge the performance of academics and their publications are flawed and fragile. Most of the rankings turn academics into invisible machines, their value established by data churned out by an invisible algorithm. For the academic careers of emerging scholars this approach is a disaster.

The points made here beg the question, what is the purpose of rankings? Our stories outlined above are a cautionary tale, warning us to beware of commercially oriented ranking systems, especially those lacking transparency. We caution against being dazzled by the so-called stars and urge academics to remain true to our purpose. These rankings undermine the academy’s foundations ‒ and by extrapolation ‒ the university mission. In their creation of a new truth, these calculative practices tell the story of the emperor’s new clothes.

Seven key global ranking reports to know (in alphabetical order):

• CWTS Leiden Rankings
• Shanghai Rankings (ARWU)
• Times Higher Education World University Rankings
• Times Higher Education Impact Rankings
• QS World University Rankings
• QS World University Rankings by Subject
• US News & World Report: Best University Rankings

Emeritus Professor James Guthrie AM, Professor of Accounting, Macquarie Business School

Professor John Dumay, Professor of Accounting, Macquarie Business School

 

 

 

[1] Parker, L., Martin-Sardesai, A., and Guthrie, J. 2021 “The commercialized Australian public university: An accountingized transition”, Financial Accountability and Management, https://doi.org/10.1111/faam.12310

[2] https://campusmorningmail.com.au/news/the-national-he-data-collection-too-little-too-late/

[3] https://campusmorningmail.com.au/news/global-university-rankings-are-a-menace/.

[4] See https://scholar.google.com/citations?hl=en&user=zKFxle4AAAAJ).

[5] See https://research.com/u/john-dumay . His highest cited paper is  S Qu and J Dumay, “The Qualitative Research Interview,” Qualitative Research in Accounting and Management 8 (3) 2011

[6] See https://research.com/u/james-guthrie The other Guthrie is the co-author of “Human Resource management and labour productivity” does industry matter?” Academy of Management Journal (2005)

[7]See  https://www.theaustralian.com.au/special-reports/how-we-identify-australias-top-researchers-and-research-institutions/news-story/8c965b6ba9849c1726038402688292e9

[8] We are proud to be one of the researchers mentioned in the top 1%. Full story and methodology found here https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000918

[9] https://www.elsevier.com/research-intelligence/university-rankings-guide