plus High Fives: universities researching above world standard
and where research was not rated
Stone (age) fruit
Ignore ERA, the really big research news this morning is from Tao Su at the Chinese Academy of Science and colleagues who report in Nature Scientific Reports that there were peaches in China before people. Gosh!
ERA advances excellence
Today’s release of the third Excellence for Research in Australia report demonstrates how the assessment process has lifted the quality of the national research effort according to Australian Research Council chair Aidan Byrne.
“With three ERA reports we can now see trends in research. The total number of research units evaluated is up 6 per cent and the averages are going up. There is a shift to better,” he told CMM.
Professor Byrne also adamantly asserted the quality of the process, rejecting claims that the ARC had acted against universities for attempting to manipulate the assessment process.
However Professor Byrne confirmed that performance data submitted by some universities is reported as “not rated.” He said these institutions are Central Queensland U, the University of Tasmania, the University of Newcastle, Victoria University, Edith Cowan U, Wollongong U and RMIT.
But he strongly rejected suggestions that any institution had structured submissions on research performance to game the system.
“A ‘not rated’ rating is no indication of a misdemeanour. … I do not think that any institutions acted with an intent to deceive.”
“The tendency in universities is to align ERA submissions to their academic units, the problem is that this can be at odds with our methodology. Some universities submitted administrative unit data instead of by methodological units, which is where methodologies emerge and where review panels cannot reliably form a rating,” he said.
If any institution had moved citations among research areas to “engineer an outcome” it would have “stood out like a sore thumb,” he added.
Professor Byrne said the problem involved 29 research units of evaluation among thousands.
While ERA 15 is not based on significant amounts of new data from the previous edition there is a comprehensive presentation of performance on a range of factors from esteem to income.
And there is more to come, today’s release is volume one – next year volume two will include data on commercialisation activities plus new information on a range of factors including gender. It will use a longitudinal analysis drawing on data from the three ERA exercises. Professor Byrne adds that this resource is important to the emerging emphasis on research impact. “We need to see if we can tell stories of connection with business.”
He adds that the emerging focus on innovation as a national driver makes an ERA IV essential.
“A high quality research environment is all about innovation,” he adds.
Golden era
This morning’s ERA is as complete a guide to the state of research in Australian universities as is possible. Certainly there is much in the methodology that is debatable, notably in the construction of the five merit bands and the factors they are based on.
And science policy experts question the overall assumption of comparing Australian research with what occurs across the world when the only comparators worth aspiring to are Western Europe and North America. However for all who are not adept in the dark arts of research metrics arguing over inputs and weightings is a path to madness and the ARC makes an overall case that its findings are fairly based and clearly constructed.
And what they show is that an ever-growing investment in research is delivering dividends, at least on the assumption that outputs are a measure of quality. Across the three ERAs, 2010, 2012 and 2015 the research productivity of Australian academics has increased at an astonishing rate. In 2003 there were 22 000 indexed journal articles, in 2013 there were 50 000. The RC being the agency it is there are all sorts of moderating qualifications, but analyse it how you will this is a doubling of output.
Onya Aidan
ERA 15 is an extraordinary achievement, an exercise in accountability that details what universities are doing with billions upon billions of the Australian peoples’ money. As such ARC chief Aidan Byrne and his colleagues have done the country a service based on intellect, energy and extraordinary effort.
Open data drives growth
If data is the energy source of the innovation age the ARC can power research productivity by opening access to the vast amounts of information it now holds on years of research output and achievements. The way to use it best is to make all the data sets open to all comers so that researchers can search for patterns and ideas that will advance productivity. Yes hacks would use the access to create (cue evil laughter) league tables but with full access they would be comprehensive and contestable. Make it so ARC.
High Fives
The ARC loathes league tables, especially lists of what universities score by discipline. Who says one five (“well above world standard”) is better than a four (“above world standard”) and two threes (“at world standard”) officials argue. Fair enough – but universities talk up their scores and the higher their number of fives the happier they are to announce them. As a rough guide to which institution rates where CMM has looked at the performance of all universities per discipline cluster and cites below every university which scores more than three “well above world standard” for fields of research within discipline clusters.
Mathematical sciences: Adelaide, Melbourne, UNSW. Each institution rated a five in three of the seven fields of research.
Physical sciences: ANU, Griffith, La Trobe, Macquarie, Melbourne, Monash, Queensland. Swinburne and Sydney, UWA all had fives in three or more fields. The University of Sydney had a perfect score, rating five in all eight fields of research
Chemical sciences: Griffith, Monash, Melbourne, UNSW, Queensland, Sydney, UTS, Wollongong. The University of Wollongong rated a five in seven of the nine fields of research. It does not research in the other two, theoretical and computational chemistry and other chemical sciences.
Earth sciences: ANU, UNSW, Tasmania, Wollongong. All of these institutions were well above world standard in three of the eight fields of research.
Environmental sciences: UWA, ANU, James Cook, Canberra and UoQ. The University of Western Australia had four ratings of five and does not research in the remaining field, “other environmental sciences”.
Biological sciences: ANU, La Trobe, Queensland, UWA, Macquarie, Monash, Adelaide, Newcastle, Tasmania. There are ten fields of research in this group. ANU and La Trobe both had a five in seven a four in another and do not research in two others.
Agriculture and veterinary sciences: Some 41 institutions conduct research in this discipline cluster but just four of them manage three or better top ratings in any of the nine fields of research. La Trobe leads with four, followed by Southern Cross, UNE and Tasmania with three each.
Information and computing sciences: Once again Australia underpeforms, with 41 institutions involved but just UNSW and Sydney (three each) having a significant “well above world standard” presence.
Engineering: This is a big field, with 17 fields of research. However, no Australian institution is strong in even half of them, UoQ leads with seven, followed by Monash, the Uni SA, and Melbourne with six each. They are followed by Adelaide and UNSW (five each) and Wollongong and RMIT (four each).
Technology: Another weak area for Australia, involving 41 institutions but only three researching disciplines at above world standard, Queensland (four) Monash (three) and QUT (three).
Medical and Health Sciences: Where the money goes and where universities pitch the strongest community service case. There are 19 research fields and Sydney leads with a five rating in 15 of them. It is followed by Adelaide and Melbourne with 13 each and then U0Q with 12, Monash and Newcastle with ten. Deakin (seven) leads ANU and UniSA (both six) with La Trobe, UNSW and Tasmania all being well above world standard in five research areas.
Built Environment and Design: No university scored three or more fives in the seven fields of research. In fact the only institutions with any top rated research are Melbourne (in two areas) and RMIT and UNSW (one each).
Education: Some 41 universities have researchers in education and the report card for almost of them states average performances. The most common ERA rating across the board is three, (world standard). The only exceptions are QUT with a single five rating, UoQ with two and the stand-out University of Melbourne with an above world standard rating for all four of the areas it researches.
Economics: With 41 institutions submitting research output to the ARC UNSW rated fives in all four of the research fields it works in. The University of Melbourne and UTS had top scores in three areas. ANU, Macquarie and UoQ scored one each, although Queensland did well with three “above world standards” – making for a world quality performance in all research areas.
Commerce, management, tourism and services: Melbourne is the outstanding performer, rating well above world standard in all of the five fields it is active in. UNSW is also strong with four “well above world standards” one “above “world standard” and one “at world standard”. UoQ did well with three top ratings and three “above world standards”.
Human society: ANU leads with four units rated five, followed by UoQ with three.
Psychology and cognitive sciences: This is a mediocre national performance. With 41 universities researching psychology, none had three or more above world class research areas and just ten had one or two. The leaders are ACU, CQU, UNSW, UoQ and UWA which all scores fives for two areas they are active in.
Law and legal studies: With just two relevant fields of research ANU, Monash, Melbourne, UNSW and Sydney all had perfect scores.
Creative arts and writing: Australian researchers did not cover themselves in glory in this area with just two universities having any research ratings above world class, Monash for visual arts and crafts and UNSW for creative writing, media studies and art theory. Solid performers include Melbourne, UoQ, Sydney with all fields rated at or above world standard.
Language, communication and culture: Melbourne leads with four research areas rated five, followed by UoQ and Sydney with three each. Opponents of media studies among op editors and leader writers will be disappointed that of 18 universities researching it only one is below the three “world standard” level.
History and Archaeology: Only ANU and Sydney have three research areas rated five. The University of Melbourne, which has just received a $10m gift to advance the study of history will surely be pleased that the two “well above world standards” it did score are for history and archaeology and historical studies categories.
Philosophy and religious studies: Sydney is alone in rating three top ranks but Monash, Melbourne and UoQ were are at the front of the second rank.
Not rated
So what research areas at which universities were “not rated” by Research Evaluation Committees due to “coding issues”?
The list is:
CQU: agriculture
Edith Cowan: psychology
U of Newcastle: psychology
RMIT: other biological sciences
Victoria U: numerical and computational maths, chemical engineering, civil engineering, food sciences, medical and health science, pharmacology, medical physiology, public health, psychology
UTas: psychology, other psychology
Uni of Wollongong: condensed matter physics, biochemistry, interdisciplinary engineering, other engineering, medical and health sciences, clinical sciences, nutrition, public health, applied economics, accounting, business and management, marketing and other commerce, psychology.
