BY WENDY INGMAN

 

I wanted to share my experience as a peer-review panel member for the recent National Health and Medical Research Council Investigator Grants.

I have been an NHMRC assessor on ECR/MCR fellowship panels for the past four years, this restructure is changing how we assess research excellence. We, as Australian medical researchers, are all invested in this new scheme and we need it to be successful to support our sector.

Before you ask, I have clarified with NHMRC and run this past my RAO @UniofAdelaide. This information does not breach my confidentiality agreement with NHMRC.

Apart from the teething problems (which are to be expected), this scheme has a lot of potential although it needs adequate funding. It puts quality over quantity and block funding means no more endless grant applications, leaving more time to do research.

There were five assessors on the panel – two women and three men. Diverse in expertise. We assessed 36 applications.

These applications were a mixture of EL1 and EL2s from applicants immediate post-PhD up to 10 years postdoc. This is the first big divergence from previous years. The old structure was ECF (<2 years postdoc), CDF1 (<7 years) and CDF2 (<12 years) separately.

The applications were a mixture of basic science, clinical medicine and public health. I had only assessed basic research in previous fellowship schemes, as those three streams used to be separate.

I thought I would struggle with applications that were not biomedical, but this ended up not being a problem. Basic science is already a hugely diverse set of applications so adding in other approaches was ok. All applications need to be written for a general scientific audience.

The assessment criteria were purely qualitative (for example, five = “excellent”, six = “outstanding”) – no quantitative benchmarking as in previous years, (for example, “it is expected the top 35 per cent of applications will score five or higher” in 2018 ECF category descriptors).

Impact statement was a game-changer for me. I assessed impact before looking at publications as it helped me assess the quality and contribution to science of the publications, as per category descriptor for the publication section.

In previous years, publications was mostly about number, journal, citations, authorship position etc. But now we get to read about the impact. If the publications have been pivotal to an outstanding impact, then those publications have made a quality contribution to science.

But how to assess impact? The category descriptors required “robust verifiable evidence” of the impact and of the contribution of the research to the impact. We were instructed that, “a poorly corroborated research impact or contribution to impact should receive a score of one”.

However, we were also instructed that, “the application should contain all information necessary for assessment without the need for further written or oral explanation or reference to additional documentation.”

Here’s the problem: How do you provide this evidence? If an applicant states that their publication has XXX citations, I don’t just believe it is true, I want to see the evidence. None of the 36 applications provided information that I would deem sufficient to verify their claims.

I did not think it was right to score them all a one, so I decided to try to verify them through InCites, Pubmed, Google etc. This was a really big job, the impacts were very diverse – citation metrics, awards, clinical guidelines, patents, products, policy documents.

I queried NHMRC on one of the applications I had trouble verifying. I told them I was using Google looking for evidence of the impact but had encountered a difficulty. Their advice was “take this in consideration when you mark the research”.

Once all the assessors had individually scored all applications, the scores were released to the panel. We each had the opportunity to nominate up to two applications for discussion at the telecon. I nominated two and another assessor nominated one.

There were three applications discussed at the telecon. After each had been discussed we had the opportunity to adjust our scores. Only these three applications could have scores adjusted, the other scores were considered already finalised.

At the telecon, the NHMRC official said that I should not have looked for evidence of the impacts online, that we should only consider the information provided in the application. Some of the other assessors generally had also done their own checks, with varying approaches.

I asked “If I claim to have won a Nobel Prize how would you know it was true or not”? The official went to consult with their supervisor, who confirmed that we are not supposed to check outside of the application.

I am still confused about this, the category descriptor puts an emphasis on “robust verifiable evidence” so should not the impact and contribution be … verified?

In checking all the evidence, I did find applicants who claimed impacts that could not be corroborated. Some were not verifiable but still might be true, in some cases though there was evidence to the contrary of what was claimed.

Every year I find a small proportion of applicants who I flag due to misleading statements. An aspect of being a peer-reviewer is to do what we can to ensure integrity of the science we assess.

I think the category descriptors rightly put the emphasis on “verifiable evidence”, but how this evidence should be presented in the application is not clear. I hope there is more guidance on this for the next round.

The majority of applicants claimed a single impact, usually ‘knowledge’. Some applications claimed multiple impacts across knowledge, health, etc, these were difficult to assess. We were advised that multiple impacts should not be additive, but assessed holistically on balance.

With multiple impacts, applicants often would make vague claims with insufficient detail, as there was not the space in the 2000-character limit to properly explain the impacts. This might be less problematic at the leader level, where multiple impacts might fit together better.

Knowledge gain: quite a few applicants used this section to put track record components that did not really address the category descriptors, at the expense of explaining key elements like a well-justified hypothesis or addressing an area of critical importance.

Relative to opportunity: this is a problem. It is very difficult for assessors to score applicants with one year postdoc experience alongside applicants with ten years of experience. Career trajectories are not linear and each phase has its own challenges.

In the last three fellowship rounds, I assessed CDF1s for two years then did ECFs the following year and really struggled with changing my expectations. I had to go back and do my benchmarking from scratch so I could find the outstanding applications within the ECF applications.

If the scheme aims to support newly graduated researchers they will need to be assessed as a group with a peer-review panel which only reads these applications.

MD/PhDs were quite dominant in the top scores. These applications were not just in the clinical stream but also in basic science and public health. They scored well in impact, publications and leadership.

I read some applications that were in my opinion outstanding/exceptional, unfortunately none of the 36 applications were ultimately successful. There was not enough agreement in scores across the assessors, and they all ended up falling short of the funding cut-off.

At the time of assessment, I did not know what the funding cut off would be or that there would be no normalisation between panels. Assessors need to be aware of this – the top ranked applicant within the panel does not necessarily get funded.

Career disruptions: this was very prominent in the application. Of the top five applicants from my panel, three were women, two of whom had substantial CDs. We still have problems with this, but in the fellowship panels I have been on I feel that CDs have been taken into consideration.

Overall, I think this is will be a good funding scheme. Yes, there are some problems that will take time to sort out. I like the impact section, and the shorter applications get right to the point – quality science contributing to improving health outcomes.

The main problem here is that there is not enough money in the pool to support the Australian research community. The scheme highlights the inadequacy, but this is not a new problem. The sector is rapidly losing critical mass, our workforce has shrunk by 20 per cent in the last five years.

I support @TheASMR1 call for immediate injection of $350 million to the NHMRC as stage one of a stepped strategy where NHMRC+MRFF investment is increased to 3 per cent of total health expenditure. Every $1 invested in the NHMRC returns $3.20 in health and economic benefits.

That’s all from me, I hope what I have put here has been helpful. Interested to hear others thoughts, we are in this together and constructive discussion is very much needed.

Associate Professor Wendy Ingman, Adelaide Medical School, The Queen Elizabeth Hospital, University of Adelaide. Wendy’s research is supported by a fellowship from The Hospital Research Foundation.