Measurements and rewards influence behaviour. They not only influence what is performed, but also what is not.
Key performance indicators are often used to measure the performance of employees; they prioritise activities and drive behaviour. The measures, rewards and potential of future grant successes that funders of research and their employers put into the research system also influence behaviour.
Large and direct influencers can overshadow the needs of smaller research funders and those without substantive researcher or institution rewards. As Part 1 of Measuring Research Impact highlights, different funders have different goals, needs and measures of success. Part 1 also highlights the need to review and change the approach to measurement, with the poll showing that current metrics impede or distract from translation.
[poll id=”24″]
Whilst current metrics are critical in maintaining an internationally competitive academic system and in sustaining a $21—22 billion export industry, the balance of measures may not always align with the key goals or expectations of other funders.
Many funders missions are to support research and translation for health benefits; not to support the export of education. The balance of current metrics and what is deemed as not important for some funders can sometimes impede translational and activities that are important for others. For instance, I am aware of an organisation whose researchers have questioned the continuation of quality systems because “the main funders of research don’t care”. These quality systems, however, are important in the translation and service activities of the organisation. This is also interesting given the high percentage of irreproducible results published in scientific journals.
Some key research activities critical to support translation are also not easily publishable, and if they are, rarely attract key journals and citations. This can result in ‘boring’ research not being undertaken, thus impeding translation. Unbalanced metrics can also support a cyclical focus on ‘discovering the next big thing’, but not following through with the discoveries. This approach meets the organisational goals of universities as the results are publishable, but all this being at the expense of community benefiting from research outputs and outcomes.
Measuring intellectual property outputs can also be used, but again those using simple measures need to be aware of potential flaws. Measuring the number of patents filed for example could drive the filing of poor quality patents and down the track the costs of maintaining unnecessary patents.
Similarly measuring the number of start-up companies could drive the establishment of numerous companies who don’t survive long past the measurement phase and also lead to innovations moving into commercial vehicles inappropriately or too early and encourage deals with the ‘wrong’ investors (i.e. “the deal with the devil”).
The Global Innovation index uses 81 measures to review inputs and outcomes from research. One key measure is the innovation efficiency ratio – the ability to turn inputs into outcomes. The 2017 results have recently been released and our previous post highlights areas for Australia to improve. Whilst these measures help identify system failings and opportunities we need to develop solutions and associated metrics that are useful in real-time.
This post seeks to encourage a discussion around balance; to consider the needs of all funders and those who support research, including the community. and the potential of measuring progress and accepting failure as a positive research outcome leading to research efficiencies and effective funding.
Importantly we all rely on the quality of a sustainable academic research system where education and blue-sky research underpins the ability to foster innovation and future translational outcomes. Much of this is also predominantly supported by tax payers, whose expectations from research expenditure may more closely align with the treatment or prevention of diseases. We need to ensure that our whole funding, measurement and reward ecosystem is supported and balanced so that we can maximise potential benefits when they exist.
Join the conversation at NFMRI’s third annual conference, “Philanthropy: Creating Impact and Dancing with Elephants”. The conference will take place on the 21st and 22nd of November 2017 at the Australian National Maritime Museum in Sydney.
